The Design and Implementation of the FreeBSD Operating System

• | Table of Contents
• | Index

By Marshall Kirk McKusick, George V. Neville-Neil
Publisher | : Addison Wesley
Pub Date | : August 02, 2004
ISBN | : 0-201-70245-2
Pages | : 720

As in earlier Addison-Wesley books on the UNIX-based BSD operating system, Kirk McKusick and George Neville-Neil deliver here the most comprehensive, up-to-date, and authoritative technical information on the internal structure of open source FreeBSD. Readers involved in technical and sales support can learn the capabilities and limitations of the system; applications developers can learn effectively and efficiently how to interface to the system; system administrators can learn how to maintain, tune, and configure the system; and systems programmers can learn how to extend, enhance, and interface to the system.

The authors provide a concise overview of FreeBSD's design and implementation. Then, while explaining key design decisions, they detail the concepts, data structures, and algorithms used in implementing the systems facilities. As a result, readers can use this book as both a practical reference and an in-depth study of a contemporary, portable, open source operating system.

This book:

  • Details the many performance improvements in the virtual memory system

  • Describes the new symmetric multiprocessor support

  • Includes new sections on threads and their scheduling

  • Introduces the new jail facility to ease the hosting of multiple domains

  • Updates information on networking and interprocess communication

Already widely used for Internet services and firewalls, high-availability servers, and general timesharing systems, the lean quality of FreeBSD also suits the growing area of embedded systems. Unlike Linux, FreeBSD does not require users to publicize any changes they make to the source code.


• | Table of Contents
• | Index

The Design and Implementation of the FreeBSD Operating System
By Marshall Kirk McKusick, George V. Neville-Neil
Publisher | : Addison Wesley
Pub Date | : August 02, 2004
ISBN | : 0-201-70245-2
Pages | : 720

Copyright
Preface
 | UNIX-like Systems
 | Berkeley Software Distributions
 | Material Covered in this Book
 | Use by Computer Professionals
 | Use in Courses on Operating Systems
 | Organization
 | Getting BSD
 | Acknowledgments
 | References
About the Authors
Part I. Overview
 | Chapter 1. History and Goals
 |  | Section 1.1. History of the UNIX System
 |  | Section 1.2. BSD and Other Systems
 |  | Section 1.3. The Transition of BSD to Open Source
 |  | Section 1.4. The FreeBSD Development Model
 |  | References
 | Chapter 2. Design Overview of FreeBSD
 |  | Section 2.1. FreeBSD Facilities and the Kernel
 |  | Section 2.2. Kernel Organization
 |  | Section 2.3. Kernel Services
 |  | Section 2.4. Process Management
 |  | Section 2.5. Memory Management
 |  | Section 2.6. I/O System
 |  | Section 2.7. Devices
 |  | Section 2.8. Filesystems
 |  | Section 2.9. Network Filesystem
 |  | Section 2.10. Terminals
 |  | Section 2.11. Interprocess Communication
 |  | Section 2.12. Network Communication
 |  | Section 2.13. Network Implementation
 |  | Section 2.14. System Operation
 |  | Exercises
 |  | References
 | Chapter 3. Kernel Services
 |  | Section 3.1. Kernel Organization
 |  | Section 3.2. System Calls
 |  | Section 3.3. Traps and Interrupts
 |  | Section 3.4. Clock Interrupts
 |  | Section 3.5. Memory-Management Services
 |  | Section 3.6. Timing Services
 |  | Section 3.7. User, Group, and Other Identifiers
 |  | Section 3.8. Resource Services
 |  | Section 3.9. System-Operation Services
 |  | Exercises
 |  | References
Part II. Processes
 | Chapter 4. Process Management
 |  | Section 4.1. Introduction to Process Management
 |  | Section 4.2. Process State
 |  | Section 4.3. Context Switching
 |  | Section 4.4. Thread Scheduling
 |  | Section 4.5. Process Creation
 |  | Section 4.6. Process Termination
 |  | Section 4.7. Signals
 |  | Section 4.8. Process Groups and Sessions
 |  | Section 4.9. Jails
 |  | Section 4.10. Process Debugging
 |  | Exercises
 |  | References
 | Chapter 5. Memory Management
 |  | Section 5.1. Terminology
 |  | Section 5.2. Overview of the FreeBSD Virtual-Memory System
 |  | Section 5.3. Kernel Memory Management
 |  | Section 5.4. Per-Process Resources
 |  | Section 5.5. Shared Memory
 |  | Section 5.6. Creation of a New Process
 |  | Section 5.7. Execution of a File
 |  | Section 5.8. Process Manipulation of Its Address Space
 |  | Section 5.9. Termination of a Process
 |  | Section 5.10. The Pager Interface
 |  | Section 5.11. Paging
 |  | Section 5.12. Page Replacement
 |  | Section 5.13. Portability
 |  | Exercises
 |  | References
Part III. I/O System
 | Chapter 6. I/O System Overview
 |  | Section 6.1. I/O Mapping from User to Device
 |  | Section 6.2. Character Devices
 |  | Section 6.3. Disk Devices
 |  | Section 6.4. Descriptor Management and Services
 |  | Section 6.5. The Virtual-Filesystem Interface
 |  | Section 6.6. Filesystem-Independent Services
 |  | Section 6.7. Stackable Filesystems
 |  | Exercises
 |  | References
 | Chapter 7. Devices
 |  | Section 7.1. Device Overview
 |  | Section 7.2. The GEOM Layer
 |  | Section 7.3. The CAM Layer
 |  | Section 7.4. The ATA Layer
 |  | Section 7.5. Device Configuration
 |  | Exercises
 |  | References
 | Chapter 8. Local Filesystems
 |  | Section 8.1. Hierarchical Filesystem Management
 |  | Section 8.2. Structure of an Inode
 |  | Section 8.3. Naming
 |  | Section 8.4. Quotas
 |  | Section 8.5. File Locking
 |  | Section 8.6. Soft Updates
 |  | Section 8.7. Filesystem Snapshots
 |  | Section 8.8. The Local Filestore
 |  | Section 8.9. The Berkeley Fast Filesystem
 |  | Exercises
 |  | References
 | Chapter 9. The Network Filesystem
 |  | Section 9.1. History and Overview
 |  | Section 9.2. NFS Structure and Operation
 |  | Section 9.3. Techniques for Improving Performance
 |  | Exercises
 |  | References
 | Chapter 10. Terminal Handling
 |  | Section 10.1. Terminal-Processing Modes
 |  | Section 10.2. Line Disciplines
 |  | Section 10.3. User Interface
 |  | Section 10.4. The tty Structure
 |  | Section 10.5. Process Groups, Sessions, and Terminal Control
 |  | Section 10.6. C-lists
 |  | Section 10.7. RS-232 and Modem Control
 |  | Section 10.8. Terminal Operations
 |  | Section 10.9. Other Line Disciplines
 |  | Exercises
 |  | References
Part IV. Interprocess Communication
 | Chapter 11. Interprocess Communication
 |  | Section 11.1. Interprocess-Communication Model
 |  | Section 11.2. Implementation Structure and Overview
 |  | Section 11.3. Memory Management
 |  | Section 11.4. Data Structures
 |  | Section 11.5. Connection Setup
 |  | Section 11.6. Data Transfer
 |  | Section 11.7. Socket Shutdown
 |  | Section 11.8. Local Interprocess-Communication
 |  | Exercises
 |  | References
 | Chapter 12. Network Communication
 |  | Section 12.1. Internal Structure
 |  | Section 12.2. Socket-to-Protocol Interface
 |  | Section 12.3. Protocol-Protocol Interface
 |  | Section 12.4. Interface Between Protocol and Network Interface
 |  | Section 12.5. Routing
 |  | Section 12.6. Buffering and Congestion Control
 |  | Section 12.7. Raw Sockets
 |  | Section 12.8. Additional Network-Subsystem Topics
 |  | Exercises
 |  | References
 | Chapter 13. Network Protocols
 |  | Section 13.1. IPv4 Network Protocols
 |  | Section 13.2. User Datagram Protocol (UDP)
 |  | Section 13.3. Internet Protocol (IP)
 |  | Section 13.4. Transmission Control Protocol (TCP)
 |  | Section 13.5. TCP Algorithms
 |  | Section 13.6. TCP Input Processing
 |  | Section 13.7. TCP Output Processing
 |  | Section 13.8. Internet Control Message Protocol (ICMP)
 |  | Section 13.9. 1Pv6
 |  | Section 13.10. Security
 |  | Exercises
 |  | References
Part V. System Operation
 | Chapter 14. Startup and Shutdown
 |  | Section 14.1. Overview
 |  | Section 14.2. Bootstrapping
 |  | Section 14.3. Kernel Initialization
 |  | Section 14.4. Kernel Module Initialization
 |  | Section 14.5. User-Level Initialization
 |  | Section 14.6. System Operation
 |  | Exercises
 |  | References
Glossary
FreeBSD Kernel Internals on Video
Advanced FreeBSD Course on Video
CSRG Archive CD-ROMs
History of UNIX at Berkeley
BSD Daemon Shirts
Index


Copyright

UNIX is a registered trademark of X/Open in the United States and other countries. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and Addison-Wesley was aware of a trademark claim, the designations have been printed with initial capital letters or in all capitals.

The authors and publisher have taken care in the preparation of this book, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of the use of the information or programs contained herein.

The publisher offers discounts on this book when ordered in quantity for bulk purchases and special sales. For more information, please contact:

U.S. Corporate and Government Sales
(800) 382-3419
corpsales@pearsontechgroup.com

For sales outside of the U.S., please contact:

International Sales
(317) 581-3793
international@pearsontechgroup.com

Visit Addison-Wesley on the Web: www.awprofessional.com

Library of Congress Cataloging-in-Publication Data

McKusick, Marshall Kirk.
The design and implementation of the FreeBSD operating system / Marshall
Kirk McKusick, George V. Neville-Neil.
p. cm.
Includes bibliographical references and index.
ISBN 0-201-70245-2 (hc : alk. paper)
1. FreeBSD. 2. Free computer software. 3. Operating systems
(Computers) I. Neville-Neil, George V. II. Title.

QA76.76.O63M398745 2004
005.3--dc22 2004010590

Copyright © 2005 by Pearson Education, Inc.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior consent of the publisher. Printed in the United States of America. Published simultaneously in Canada.

For information on obtaining permission for use of material from this work, please submit a written request to:

Pearson Education, Inc.
Rights and Contracts Department
75 Arlington Street, Suite 300
Boston, MA 02116
Fax: (617) 848-7047

Text printed on recycled and acid-free paper.

1 2 3 4 5 6 7 8 9 10-CRW-0807060504

First Printing, July 2004

Dedication

This book is dedicated to the BSD community.

Without the contributions of that community's members, there would be nothing about which to write.


Preface

This book follows the earlier authoritative and full-length descriptions of the design and implementation of the 4.3BSD and 4.4BSD versions of the UNIX system developed at the University of California at Berkeley. Since the final Berkeley release in 1994, several groups have continued development of BSD. This book details FreeBSD, the system with the largest set of developers and the most widely distributed releases. Although the FreeBSD distribution includes nearly 1000 utility programs in its base system and nearly 10,000 optional utilities in its ports collection, this book concentrates almost exclusively on the kernel.


UNIX-like Systems

UNIX-like systems include the traditional vendor systems such as Solaris and HP-UX; the Linux-based distributions such as Red Hat, Debian, Suse, and Slackware; and the BSD-based distributions such as FreeBSD, NetBSD, OpenBSD, and Darwin. They run on computers ranging from laptops to the largest supercomputers. They are the operating system of choice for most multiprocessor, graphics, and vector-processing systems, and are widely used for the original purpose of timesharing. The most common platform for providing network services (from FTP to WWW) on the Internet, they are collectively the most portable operating system ever developed. This portability is due partly to their implementation language, C [Kernighan & Ritchie, 1989] (which is itself a widely ported language), and partly to the elegant design of the system.

Since its inception in 1969 [Ritchie & Thompson, 1978], the UNIX system has developed in several divergent and rejoining streams. The original developers continued to advance the state of the art with their Ninth and Tenth Edition UNIX inside AT&T Bell Laboratories, and then their Plan 9 successor to UNIX. Meanwhile, AT&T licensed UNIX System V as a product before selling it to Novell. Novell passed the UNIX trademark to X/OPEN and sold the source code and distribution rights to Santa Cruz Operation (SCO). Both System V and Ninth Edition UNIX were strongly influenced by the Berkeley Software Distributions produced by the Computer Systems Research Group (CSRG) of the University of California at Berkeley. The Linux operating system, although developed independently of the other UNIX variants, implements the UNIX interface. Thus, applications developed to run on other UNIX-based platforms can be easily ported to run on Linux.


Berkeley Software Distributions

The distributions from Berkeley were the first UNIX-based systems to introduce many important features including the following:

  • Demand-paged virtual-memory support

  • Automatic configuration of the hardware and I/O system

  • A fast and recoverable filesystem

  • The socket-based interprocess-communication (IPC) primitives

  • The reference implementation of TCP/IP

The Berkeley releases found their way into the UNIX systems of many vendors and were used internally by the development groups of many other vendors. The implementation of the TCP/IP networking protocol suite in 4.2BSD and 4.3BSD, and the availability of those systems, played a key role in making the TCP/IP networking protocol suite a world standard. Even the non-UNIX vendors such as Microsoft have adopted the Berkeley socket design in their Winsock IPC interface.

The BSD releases have also been a strong influence on the POSIX (IEEE Std 1003.1) operating-system interface standard, and on related standards. Several features—such as reliable signals, job control, multiple access groups per process, and the routines for directory operations—have been adapted from BSD for POSIX.

Early BSD releases contained licensed UNIX code, thus requiring recipients to have an AT&T source license to be able to obtain and use BSD. In 1988, Berkeley separated its distribution into AT&T licensed and freely redistributable code. The freely redistributable code was licensed separately and could be obtained, used, and redistributed by anyone. The final freely redistributable 4.4BSD-Lite2 release from Berkeley in 1994 contained nearly the entire kernel and all the important libraries and utilities.

Two groups, NetBSD and FreeBSD, sprang up in 1993 to begin supporting and distributing systems built from the freely redistributable releases being done by Berkeley. The NetBSD group emphasized portability and the minimalist approach, porting the systems to nearly forty platforms and pushing to keep the system lean to aid embedded applications. The FreeBSD group emphasized maximal support for the PC architecture and pushed to ease installation for, and market their system to, as wide an audience as possible. In 1995, the OpenBSD group split from the NetBSD group to develop a distribution that emphasized security. Over the years there has been a healthy competition among the BSD distributions, with many ideas and much code flowing between them.


Material Covered in this Book

This book is about the internal structure of the FreeBSD 5.2 kernel and about the concepts, data structures, and algorithms used in implementing FreeBSD's system facilities. Its level of detail is similar to that of Bach's book about UNIX System V [Bach, 1986]; however, this text focuses on the facilities, data structures, and algorithms used in the FreeBSD variant of the UNIX operating system. The book covers FreeBSD from the system-call level down—from the interface to the kernel to the hardware itself. The kernel includes system facilities, such as process management, virtual memory, the I/O system, filesystems, the socket IPC mechanism, and network protocol implementations. Material above the system-call level— such as libraries, shells, commands, programming languages, and other user interfaces—is excluded, except for some material related to the terminal interface and to system startup. Following the organization first established by Organick's book about Multics [Organick, 1975], this book is an in-depth study of a contemporary operating system.

Where particular hardware is relevant, the book refers to the Intel Personal Computer (PC) architecture. Because FreeBSD has emphasized development on the PC, that is the architecture with the most complete support, so it provides a convenient point of reference.


Use by Computer Professionals

FreeBSD is widely used to support the core infrastructure of many companies worldwide. Because it can be built with a small footprint, it is also seeing increased use in embedded applications. The licensing terms of FreeBSD do not require the distribution of changes and enhancements to the system. The licensing terms of Linux require that all changes and enhancements to the kernel be made available in source form at minimal cost. Thus, companies that need to control the distribution of their intellectual property build their products using FreeBSD.

This book is of direct use to the professionals who work with FreeBSD systems. Individuals involved in technical and sales support can learn the capabilities and limitations of the system; applications developers can learn how to effectively and efficiently interface to the system; system administrators without direct experience with the FreeBSD kernel can learn how to maintain, tune, and configure the system; and systems programmers can learn how to extend, enhance, and interface to the system.

Readers who will benefit from this book include operating-system implementors, system programmers, UNIX application developers, administrators, and curious users. The book can be read as a companion to the source code of the system, falling as it does between the manual pages and the code in detail of treatment. But this book is neither exclusively a UNIX programming manual nor a user tutorial (for a tutorial, see Libes & Ressler [1988]). Familiarity with the use of some version of the UNIX system (see, for example, Stevens [1992]) and with the C programming language (see, for example, Kernighan & Ritchie [1989]) would be extremely useful.


Use in Courses on Operating Systems

This book is suitable for use as a reference text to provide background for a primary textbook in a first-level course on operating systems. It is not intended for use as an introductory operating-system textbook; the reader should have already encountered terminology such as memory management, process scheduling, and I/O systems [Silberschatz et al., 2002]. Familiarity with the concepts of network protocols [Comer, 2000; Stallings, 2000; Tanenbaum, 2003] will be useful for understanding some of the later chapters.

This book can be used in combination with a copy of the FreeBSD system for more advanced operating systems courses. Students' assignments can include changes to, or replacements of, key system components such as the scheduler, the paging daemon, the filesystems, thread signalling, various networking layers, and I/O management. The ability to load, replace, and unload modules from a running kernel allows students to experiment without the need to compile and reboot the system. By working with a real operating system, students can directly measure and experience the effects of their changes. Because of the intense peer review and insistence on well-defined coding standards throughout its 25-year lifetime, the FreeBSD kernel is considerably cleaner, more modular, and thus easier to understand and modify than most software projects of its size and age.

Exercises are provided at the end of each chapter. The exercises are graded into three categories indicated by zero, one, or two asterisks. The answers to exercises that carry no asterisks can be found in the text. Exercises with a single asterisk require a step of reasoning or intuition beyond a concept presented in the text. Exercises with two asterisks present major design projects or open research questions.


Organization

This text discusses both philosophical and design issues, as well as details of the actual implementation. Often, the discussion starts at the system-call level and descends into the kernel. Tables and figures are used to clarify data structures and control flow. Pseudocode similar to the C language displays algorithms. Boldface font identifies program names and filesystem pathnames. Italics font introduces terms that appear in the glossary and identifies the names of system calls, variables, routines, and structure names. Routine names (other than system calls) are further identified by the name followed by a pair of parenthesis (e.g., malloc() is the name of a routine, whereas argv is the name of a variable).

The book is divided into five parts, organized as follows:

  • Part I, Overview Three introductory chapters provide the context for the complete operating system and for the rest of the book. Chapter 1, History and Goals, sketches the historical development of the system, emphasizing the system's research orientation. Chapter 2, Design Overview of FreeBSD, describes the services offered by the system and outlines the internal organization of the kernel. It also discusses the design decisions that were made as the system was developed. Sections 2.3 through 2.14 in Chapter 2 give an overview of their corresponding chapter. Chapter 3, Kernel Services, explains how system calls are done and describes in detail several of the basic services of the kernel.

  • Part II, Processes The first chapter in this part—Chapter 4, Process Management—lays the foundation for later chapters by describing the structure of a process, the algorithms used for scheduling the execution of the threads that make up a process, and the synchronization mechanisms used by the system to ensure consistent access to kernel-resident data structures. In Chapter 5, Memory Management, the virtual-memory-management system is discussed in detail.

  • Part III, I/O System First, Chapter 6, I/O System Overview, explains the system interface to I/O and describes the structure of the facilities that support this interface. Following this introduction are four chapters that give the details of the main parts of the I/O system. Chapter 7, Devices, gives a description of the I/O architecture of the PC and describes how the I/O subsystem is managed and how the kernel initially maps out and later manages the arrival and departure of connected devices. Chapter 8, Local Filesystems, details the data structures and algorithms that implement filesystems as seen by application programs as well as how local filesystems are interfaced with the device interface described in Chapter 7. Chapter 9, The Network Filesystem, explains the network filesystem from both the server and client perspectives. Chapter 10, Terminal Handling, discusses support for character terminals and provides a description of the pseudo-terminal device driver.

  • Part IV, Interprocess Communication Chapter 11, Interprocess Communication, describes the mechanism for providing communication between related or unrelated processes. Chapters 12 and 13, Network Communication and Network Protocols, are closely related because the facilities explained in the former are implemented by specific protocols, such as the TCP/IP protocol suite, explained in the latter.

  • Part V, System Operation Chapter 14, Startup and Shutdown, discusses system startup and shutdown and explains system initialization at the process level from kernel initialization to user login.

The book is intended to be read in the order that the chapters are presented, but the parts other than Part I are independent of one another and can be read separately. Chapter 14 should be read after all the others, but knowledgeable readers may find it useful independently.

At the end of the book are a Glossary with brief definitions of major terms and an Index. Each chapter contains a Reference section with citations of related material.


Getting BSD

All the BSD distributions are available either for downloading from the net or on removable media such as CD-ROM or DVD. Information on obtaining source and binaries for FreeBSD can be obtained from http://www.FreeBSD.org. The NetBSD distribution is compiled and ready to run on most workstation architectures. For more information, contact the NetBSD Project at http://www.NetBSD.org/. The OpenBSD distribution is compiled and ready to run on a wide variety of workstation architectures and has been extensively vetted for security and reliability. For more information, visit the OpenBSD project's Web site at http://www.OpenBSD.org/.

For you dichards who actually read to the end of the preface, your reward is finding out that you can get T-shirts that are a reproduction of the original artwork drawn by John Lasseter for the cover of this book (yes, he is the John Lasseter of Pixar fame who masterminded the production of Toy Story). For further information on purchasing a shirt, visit the "History of BSD T-shirts" Web page at http://www.McKusick.com/beastie/. Other items available for sale on the site include a 32-hour introductory video course based on this book, a 40-hour advanced video course based on the FreeBSD 5.2 source code, a 2.5-hour video lecture on the history of BSD, and a 4-CD set containing all the releases and the source-control history of BSD from Berkeley. These items are described in the advertisements that follow the Index.


Acknowledgments

We extend special thanks to Natc Lawson, who provided invaluable insight on the workings of the PC architecture and FreeBSD's interface to it.

We also thank the following people who read and commented on nearly the entire book: Michael Schuster (Sun Microsystems, Inc.) and our Addison-Wesley reviewers Chris Cooper (Hewlett-Packard) and Robert Kitzberger (IBM).

We thank the following people, all of whom read and commented on early drafts of various chapters of the book: Samy Al Bahra (Kerneled.com), Dorr H. Clark (Santa Clara University), Matthew Dillon (The DragonFly BSD Project), John Dyson, Andreas Gustafsson, Poul-Henning Kamp (The FreeBSD Project), David G. Lawrence (The FreeBSD Project), Samuel Leffler, M. Warner Losh (Timing Solutions), Andre Oppermann (Internet Business Solutions AG), David I. Parfitt (independent hacker), Doug Rabson (Qube Software Ltd.), Jeffrey Roberson (The FreeBSD Project), Soren Schmidt (FreeBSD senior developer), Ken Smith (University at Buffalo CSE Department), Gregory Sutter (Daemon News), Charles P. Wright (SUNY Stony Brook), and Erez Zadok (Stony Brook University).

We are grateful to our editor, Peter Gordon, who had faith in our ability to get the book written despite several years of delays on our part and who accelerated the production when we finally had a completed manuscript. We thank all the professional people at Addison-Wesley who helped us bring the book to completion: managing editor John Fuller, editorial assistant Bernie Gaffney, production supervisor Elizabeth Ryan, and cover designer Chuti Prasertsith. We also thank the people at Stratford Publishing Services: manager of editorial services Kathy Glidden, copy editor Debbie Prato, and proofreader Hilary Farquhar. Finally we acknowledge the contributions of Jaap Akkerhuis, who designed the troff macros for the BSD books, and John Lasseter, who drew the original BSD daemon art used on the cover.

This book was produced using James Clark's implementations of pic, tbl, eqn, and groff. The index was generated by awk scripts derived from indexing programs written by Jon Bentley and Brian Kernighan [Bentley & Kernighan, 1986]. Most of the art was created with xfig. Figure placement and widow elimination were handled by the groff macros, but orphan elimination and production of even page bottoms had to be done by hand.

We encourage readers to send us suggested improvements or comments about typographical or other errors found in the book; please send electronic mail to FreeBSDbook-bugs@McKusick.COM.


References

Bach, 1986. M. J. Bach , The Design of the UNIX Operating System, Prentice-Hall, Englewood Cliffs, NJ, 1986.

Bentley & Kernighan, 1986. J. Bentley & B. Kernighan , "Tools for Printing Indexes," Computing Science Technical Report 128, AT&T Bell Laboratories, Murray Hill, NJ, 1986.

Comer, 2000. D. Comer , Internetworking with TCP/IP Volume 1, 4th ed., Prentice-Hall, Upper Saddle River, NJ, 2000.

Kernighan & Ritchie, 1989. B. W. Kernighan & D. M. Ritchie , The C Programming Language, 2nd ed., Prentice-Hall, Englewood Cliffs, NJ, 1989.

Libes & Ressler, 1988. D. Libes & S. Ressler , Life with UNIX, Prentice-Hall, Englewood Cliffs, NJ, 1988.

Organick, 1975. E. I. Organick , The Multics System: An Examination of Its Structure, MIT Press, Cambridge, MA, 1975.

Ritchie & Thompson, 1978. D. M. Ritchie & K. Thompson , "The UNIX Time-Sharing System," Bell System Technical Journal, vol. 57, no. 6, Part 2, pp. 78-90, July-August 1978. The original version [Comm. ACM vol. 7, no. 7, pp. 365-375 (July 1974)] described the 6th edition; this citation describes the 7th edition.

Silberschatz et al., 2002. A. Silberschatz, P. Galvin, & G. Gagne , Operating System Concepts, 6th ed., John Wiley and Sons, Hoboken, NJ, 2002.

Stallings, 2000. R. Stallings , Data and Computer Communications, 6th ed., Prentice Hall, Hoboken, NJ, 2000.

Stevens, 1992. W. Stevens , Advanced Programming in the Unix Environment, Addison-Wesley, Reading, MA, 1992.

Tanenbaum, 2003. A. S. Tanenbaum , Computer Networks, 4th ed., Prentice-Hall, Englewood Cliffs, NJ, 2003.


About the Authors

Marshall Kirk McKusick writes books and articles, consults, and teaches classes on UNIX- and BSD-related subjects. While at the University of California at Berkeley, he implemented the 4.2BSD fast file system and was the Research Computer Scientist at the Berkeley Computer Systems Research Group (CSRG), overseeing the development and release of 4.3BSD and 4.4BSD. His particular areas of interest are the virtual-memory system and the filesystem. He earned his undergraduate degree in electrical engineering from Cornell University and did his graduate work at the University of California at Berkeley, where he received master's degrees in computer science and business administration and a doctoral degree in computer science. He has twice been president of the board of the Usenix Association, is currently a member of the editorial board of ACM's Queue magazine, and is a member of the Usenix Association, ACM, and IEEE. In his spare time, he enjoys swimming, scuba diving, and wine collecting. The wine is stored in a specially constructed wine cellar (accessible from the Web at http://www.McKusick.com/cgi-bin/readhouse) in the basement of the house that he shares with Eric Allman, his domestic partner of 25-and-some-odd years.

George V. Neville-Neil works on networking and operating system code for fun and profit and also teaches courses on various subjects related to programming. His areas of interest are code spelunking, real-time operating systems, and networking. He earned his bachelor's degree in computer science at Northeastern University in Boston, Massachusetts. He serves on the editorial board of ACM's Queue magazine and is a member of the Usenix Association, ACM, and IEEE. He is an avid bicyclist, motorcyclist, and traveler who has made San Francisco his home since 1990.


Part I: Overview

    Chapter 1. History and Goals

    Chapter 2. Design Overview of FreeBSD

    Chapter 3. Kernel Services


Chapter 1. History and Goals


1.1. History of the UNIX System

The UNIX system has been in wide use for over 30 years and has helped to define many areas of computing. Although numerous individuals and organizations have contributed (and still contribute) to the development of the UNIX system, this book primarily concentrates on the BSD thread of development.

  • Bell Laboratories, which invented UNIX

  • The Computer Systems Research Group (CSRG) at the University of California at Berkeley, which gave UNIX virtual memory and the reference implementation of TCP/IP

  • The FreeBSD project, the NetBSD project, and the OpenBSD project, which continue the work started by the CSRG

  • The Darwin operating system at the core of Apple's OS X. Darwin is based on FreeBSD.

Origins

The first version of the UNIX system was developed at Bell Laboratories in 1969 by Ken Thompson as a private research project to use an otherwise idle PDP-7. Thompson was joined shortly thereafter by Dennis Ritchie, who not only contributed to the design and implementation of the system, but also invented the C programming language. The system was completely rewritten into C, leaving almost no assembly language. The original elegant design of the system [Ritchie, 1978] and developments of the first 15 years [Ritchie, 1984a; Compton, 1985] have made the UNIX system an important and powerful operating system [Ritchie, 1987].

Ritchie, Thompson, and other early UNIX developers at Bell Laboratories had worked previously on the Multics project [Peirce, 1985; Organick, 1975], which had a strong influence on the newer operating system. Even the name UNIX is merely a pun on Multics; in areas where Multics attempted to do many tasks, UNIX tried to do only one task but do it well. The basic organization of the UNIX filesystem, the idea of using a user process for the command interpreter, the general organization of the filesystem interface, and many other system characteristics come directly from Multics.

Ideas from various other operating systems, such as the Massachusetts Institute of Technology's (MIT's) CTSS, also have been incorporated. The fork operation to create new processes comes from Berkeley's GENIE (SDS-940, later XDS-940) operating system. Allowing a user to create processes inexpensively led to using one process per command rather than commands being run as procedure calls, as is done in Multics.

Research UNIX

The first major editions of UNIX were the Research systems from Bell Laboratories. In addition to the earliest versions of the system, these systems include the UNIX Time-Sharing System, Sixth Edition, commonly known as V6, which in 1976 was the first version widely available outside of Bell Laboratories. Systems are identified by the edition numbers of the UNIX Programmer's Manual that were current when the distributions were made.

The UNIX system was distinguished from other operating systems in three important ways.

  1. It was written in a high-level language.

  2. It was distributed in source form.

  3. It provided powerful primitives normally found in only those operating systems that ran on much more expensive hardware.

Most of the system source code was written in C rather than in assembly language. The prevailing belief at the time was that an operating system had to be written in assembly language to provide reasonable efficiency and to get access to the hardware. The C language itself was at a sufficiently high level to allow it to be compiled easily for a wide range of computer hardware, without its being so complex or restrictive that systems programmers had to revert to assembly language to get reasonable efficiency or functionality. Access to the hardware was provided through assembly-language stubs for the 3 percent of the operating-system functions—such as context switching—that needed them. Although the success of UNIX does not stem solely from its being written in a high-level language, the use of C was a critical first step [Kernighan & Ritchie, 1978; Kernighan & Ritchie, 1989; Ritchie et al., 1978]. Ritchie's C language is descended [Rosler, 1984] from Thompson's B language, which was itself descended from BCPL [Richards & Whitby-Strevens, 1980]. C continues to evolve [Tuthill, 1985; ISO, 1999].

The second important distinction of UNIX was its early release from Bell Laboratories to other research environments in source form. By providing source, the system's founders ensured that other organizations would be able not only to use the system, but also to tinker with its inner workings. The ease with which new ideas could be adopted into the system always has been key to the changes that have been made to it. Whenever a new system that tried to upstage UNIX came along, somebody would dissect the newcomer and clone its central ideas into UNIX. The unique ability to use a small, comprehensible system, written in a high-level language, in an environment swimming in new ideas led to a UNIX system that evolved far beyond its humble beginnings. Though recipients of the source code had to be licensed, campuswide licenses were cheaply available to universities. Thus, many people became versed in the way that UNIX worked, setting the stage for the open-source world that would follow.

The third important distinction of UNIX was that it provided individual users with the ability to run multiple processes concurrently and to connect these processes into pipelines of commands. At the time, only operating systems running on large and expensive machines had the ability to run multiple processes, and the number of concurrent processes usually was controlled tightly by a system administrator.

Most early UNIX systems ran on the PDP-11, which was inexpensive and powerful for its time. Nonetheless, there was at least one early port of Sixth Edition UNIX to a machine with a different architecture: the Interdata 7/32 [Miller, 1978]. The PDP-11 also had an inconveniently small address space. The introduction of machines with 32-bit address spaces, especially the VAX-11/780, provided an opportunity for UNIX to expand its services to include virtual memory and networking. Earlier experiments by the Research group in providing UNIX-like facilities on different hardware had led to the conclusion that it was as easy to move the entire operating system as it was to duplicate UNIX's services under another operating system. The first UNIX system with portability as a specific goal was UNIX Time-Sharing System, Seventh Edition (V7), which ran on the PDP-11 and the Interdata 8/32 and had a VAX variety called UNIX/32V Time-Sharing, System Version 1.0 (32V). The Research group at Bell Laboratories has also developed UNIX Time-Sharing System, Eighth Edition (V8); UNIX Time-Sharing System, Ninth Edition (V9); and UNIX Time-Sharing System, Tenth Edition (V10). Their 1996 system is Plan 9.

AT&T UNIX System III and System V

After the distribution of Seventh Edition in 1978, the Research group turned over external distributions to the UNIX Support Group (USG). USG had previously distributed internally such systems as the UNIX Programmer's Work Bench (PWB), and had sometimes distributed them externally as well [Mohr, 1985].

USG's first external distribution after Seventh Edition was UNIX System III (System III) in 1982, which incorporated features of Seventh Edition, of 32V, and also of several UNIX systems developed by groups other than the Research group. Features of UNIX/RT (a real-time UNIX system) were included, as were many features from PWB. USG released UNIX System V (System V) in 1983; that system is largely derived from System III. The court-ordered divestiture of the Bell Operating Companies from AT&T permitted AT&T to market System V aggressively [Bach, 1986; Wilson, 1985].

USG metamorphosed into the UNIX System Development Laboratory (USDL), which released UNIX System V, Release 2 in 1984. System V, Release 2, Version 4 introduced paging [Jung, 1985; Miller, 1984], including copy-on-write and shared memory, to System V. The System V implementation was not based on the Berkeley paging system. USDL was succeeded by AT&T Information Systems (ATTIS), which distributed UNIX System V, Release 3, in 1987. That system included STREAMS, an IPC mechanism adopted from V8 [Presotto & Ritchie, 1985]. ATTIS was succeeded by UNIX System Laboratory (USL), which was sold to Novell in 1993. Novell passed the UNIX trademark to the X/OPEN consortium, giving the latter sole rights to set up certification standards for using the UNIX name on products. Two years later, Novell sold UNIX to The Santa Cruz Operation (SCO).

Berkeley Software Distributions

The most influential of the non-Bell Laboratories and non-AT&T UNIX development groups was the University of California at Berkeley [DiBona et al., 1999]. Software from Berkeley was released in Berkeley Software Distributions (BSD)—for example, as 4.4BSD. Berkeley was the source of the BSD name, and their distributions were the first distinct identity for the BSD operating system. The first Berkeley VAX UNIX work was the addition to 32V of virtual memory, demand paging, and page replacement in 1979 by William Joy and Ozalp Babaoglu, to produce 3BSD [Babaoglu & Joy, 1981]. The reason for the large virtual-memory space of 3BSD was the development of what at the time were large programs, such as Berkeley's Franz LISP. This memory-management work convinced the Defense Advanced Research Projects Agency (DARPA) to fund the Berkeley team for the later development of a standard system (4BSD) for DARPA's contractors to use.

A goal of the 4BSD project was to provide support for the DARPA Internet networking protocols, TCP/IP [Comer, 2000]. The networking implementation was general enough to communicate among diverse network facilities, ranging from local networks, such as Ethernets and token rings, to long-haul networks, such as DARPA's ARPANET.

We refer to all the Berkeley VAX UNIX systems following 3BSD as 4BSD, although there were really several releases: 4.0BSD, 4.1BSD, 4.2BSD, 4.3BSD, 4.3BSD Tahoe, and 4.3BSD Reno. 4BSD was the UNIX operating system of choice for VAXes from the time that the VAX first became available in 1977 until the release of System V in 1983. Most organizations would purchase a 32V license but would order 4BSD from Berkeley. Many installations inside the Bell System ran 4.1BSD (and replaced it with 4.3BSD when the latter became available). A new virtual-memory system was released with 4.4BSD. The VAX was reaching the end of its useful lifetime, so 4.4BSD was not ported to that machine. Instead, 4.4BSD ran on the newer 68000, SPARC, MIPS, and Intel PC architectures.

The 4BSD work for DARPA was guided by a steering committee that included many notable people from both commercial and academic institutions. The culmination of the original Berkeley DARPA UNIX project was the release of 4.2BSD in 1983; further research at Berkeley produced 4.3BSD in mid-1986. The next releases included the 4.3BSD Tahoe release of June 1988 and the 4.3BSD Reno release of June 1990. These releases were primarily ports to the Computer Consoles Incorporated hardware platform. Interleaved with these releases were two unencumbered networking releases: the 4.3BSD Netl release of March 1989 and the 4.3BSD Net2 release of June 1991. These releases extracted nonproprietary code from 4.3BSD; they could be redistributed freely in source and binary form to companies that and individuals who were not covered by a UNIX source license. The final CSRG release requiring an AT&T source license was 4.4BSD in June 1993. Following a year of litigation (see Section 1.3), the free-redistributable 4.4BSD-Lite was released in April 1994. The final CSRG release was 4.4BSD-Lite Release 2 in June 1995.

UNIX in the World

The UNIX system is also a fertile field for academic endeavor. Thompson and Ritchie were given the Association for Computing Machinery Turing award for the design of the system [Ritchie, 1984b]. The UNIX system and related, specially designed teaching systems—such as Tunis [Ewens et al., 1985; Holt, 1983], XINU [Comer, 1984], and MINIX [Tanenbaum, 1987]—are widely used in courses on operating systems. Linus Torvalds reimplemented the UNIX interface in his freely redistributable Linux operating system. The UNIX system is ubiquitous in universities and research facilities throughout the world, and is ever more widely used in industry and commerce.


1.2. BSD and Other Systems

The CSRG incorporated features from not only UNIX systems but from other operating systems. Many of the features of the 4BSD terminal drivers are from TENEX/TOPS-20. Job control (in concept—not in implementation) is derived from TOPS-20 and from the MIT Incompatible Timesharing System (ITS). The virtual-memory interface first proposed for 4.2BSD, and finally implemented in 4.4BSD, was based on the file-mapping and page-level interfaces that first appeared in TENEX/TOPS-20. The current FreeBSD virtual-memory system (see Chapter 5) was adapted from Mach, which was itself an offshoot of 4.3BSD. Multics has often been a reference point in the design of new facilities.

The quest for efficiency was a major factor in much of the CSRG's work. Some efficiency improvements were made because of comparisons with the proprietary operating system for the VAX, VMS [Joy, 1980; Kashtan, 1980].

Other UNIX variants have adopted many 4BSD features. AT&T UNIX System V [AT&T, 1987], the IEEE POSIX.1 standard [P1003.1, 1988], and the related National Bureau of Standards (NBS) Federal Information Processing Standard (FIPS) have adopted the following.

  • Job control (Chapter 2)

  • Reliable signals (Chapter 4)

  • Multiple file-access permission groups (Chapter 6)

  • Filesystem interfaces (Chapter 8)

The X/OPEN Group (originally consisting of only European vendors but now including most U.S. UNIX vendors) produced the X/OPEN Portability Guide [X/OPEN, 1987] and, more recently, the Spec 1170 Guide. These documents specify both the kernel interface and many of the utility programs available to UNIX system users. When Novell purchased UNIX from AT&T in 1993, it transferred exclusive ownership of the UNIX name to X/OPEN. Thus, all systems that want to brand themselves as UNIX must meet the X/OPEN interface specifications. To date, no BSD system has ever been put through the X/OPEN interface-specification tests, so none of them can be called UNIX. The X/OPEN guides have adopted many of the POSIX facilities. The POSIX.1 standard is also an ISO International Standard, named SC22 WG15. Thus, the POSIX facilities have been accepted in most UNIX-like systems worldwide.

The 4BSD socket interprocess-communication mechanism (see Chapter 11) was designed for portability and was immediately ported to AT&T System III, although it was never distributed with that system. The 4BSD implementation of the TCP/IP networking protocol suite (see Chapter 13) is widely used as the basis for further implementations on systems ranging from AT&T 3B machines running System V to VMS to embedded operating systems such as VxWorks.

The CSRG cooperated closely with vendors whose systems are based on 4.2BSD and 4.3BSD. This simultaneous development contributed to the ease of further ports of 4.3BSD and to ongoing development of the system.

The Influence of the User Community

Much of the Berkeley development work was done in response to the user community. Ideas and expectations came not only from DARPA, the principal direct-funding organization, but also from users of the system at companies and universities worldwide.

The Berkeley researchers accepted not only ideas from the user community but also actual software. Contributions to 4BSD came from universities and other organizations in Australia, Canada, Europe, Japan, and the United States. These contributions included major features, such as autoconfiguration and disk quotas. A few ideas, such as the fcntl system call, were taken from System V, although licensing and pricing considerations prevented the use of any code from System III or System V in 4BSD. In addition to contributions that were included in the distributions proper, the CSRG also distributed a set of user-contributed software.

An example of a community-developed facility is the public-domain time-zone-handling package that was adopted with the 4.3BSD Tahoe release. It was designed and implemented by an international group, including Arthur Olson, Robert Elz, and Guy Harris, partly because of discussions in the USENET news-group comp.std.unix. This package takes time-zone-conversion rules completely out of the C library, putting them in files that require no system-code changes to change time-zone rules; this change is especially useful with binary-only distributions of UNIX. The method also allows individual processes to choose rules rather than keeping one ruleset specification systemwide. The distribution includes a large database of rules used in many areas throughout the world, from China to Australia to Europe. Distributions are thus simplified because it is not necessary to have the software set up differently for different destinations, as long as the whole database is included. The adoption of the time-zone package into BSD brought the technology to the attention of commercial vendors, such as Sun Microsystems, causing them to incorporate it into their systems.


1.3. The Transition of BSD to Open Source

Up through the release of 4.3BSD Tahoe, all recipients of BSD had to first get an AT&T source license. That was because the BSD systems were never released by Berkeley in a binary-only format; the distributions always contained the complete source to every part of the system. The history of the UNIX system, and the BSD system in particular, had shown the power of making the source available to the users. Instead of passively using the system, they actively worked to fix bugs, improve performance and functionality, and even add completely new features.

With the increasing cost of the AT&T source licenses, vendors that wanted to build stand-alone TCP/IP-based networking products for the PC market using the BSD code found the per-binary costs prohibitive. So they requested that Berkeley break out the networking code and utilities and provide them under licensing terms that did not require an AT&T source license. The TCP/IP networking code clearly did not exist in 32/V and thus had been developed entirely by Berkeley and its contributors. The BSD-originated networking code and supporting utilities were released in June 1989 as Networking Release 1, the first freely redistributable code from Berkeley.

The licensing terms were liberal. A licensee could release the code modified or unmodified in source or binary form with no accounting or royalties to Berkeley. The only requirements were that the copyright notices in the source file be left intact and that products that incorporated the code include in their documentation that the product contained code from the University of California and its contributors. Although Berkeley charged a $1000 fee to get a tape, anyone was free to get a copy from somebody who already had it. Indeed, several large sites put it up for anonymous FTP shortly after it was released. Though the code was freely available, several hundred organizations purchased tapes, which helped to fund the CSRG and encouraged further development.

Networking Release 2

With the success of the first open-source release, the CSRG decided to see how much more of BSD they could spring free. Keith Bostic led the charge by soliciting people to rewrite the UNIX utilities from scratch based solely on their published descriptions. Their only compensation would be to have their name listed among the Berkeley contributors next to the name of the utility that they rewrote. The contributions started slowly and were mostly for the trivial utilities. But as the list of completed utilities grew, and Bostic continued to hold forth for contributions at public events such as Usenix, the rate of contributions continued to grow. Soon the list crossed 100 utilities, and within 18 months nearly all the important utilities and libraries had been rewritten.

The kernel proved to be a bigger task because it could not easily be rewritten from scratch. The entire kernel was reviewed, file by file, removing code that had originated in the 32/V release. When the review was completed, there were only six remaining kernel files that were still contaminated and that could not be trivially rewritten. While consideration was given to rewriting those six files so that a complete kernel could be released, the CSRG decided to release just the less controversial set. The CSRG sought permission for the expanded release from folks higher up in the university administration. After much internal debate and verification of the methods used for detecting proprietary code, the CSRG was given permission to do the release.

The initial thought was to come up with a new name for the second freely redistributable release. However, getting a new license written and approved by the university lawyers would have taken many months. So, the new release was named Networking Release 2, since that could be done with just a revision of the approved Networking Release 1 license agreement. This second greatly expanded freely redistributable release began shipping in June 1991. The redistribution terms and cost were the same as the terms and cost of the first networking release. As before, several hundred individuals and organizations paid the $1000 fee to get the distribution from Berkeley.

Closing the gap from the Networking Release 2 distribution to a fully functioning system did not take long. Within six months of the release, Bill Jolitz had written replacements for the six missing files. He promptly released a fully compiled and bootable system for the 386-based PC architecture in January 1992, which he called 386/BSD. Jolitz's 386/BSD distribution was done almost entirely on the net. He simply put it up for anonymous FTP and let anyone who wanted it download it for free. Within weeks he had a huge following.

Unfortunately, the demands of keeping a full-time job meant that Jolitz could not devote the time needed to keep up with the flood of incoming bug fixes and enhancements to 386/BSD. So within a few months of the release of 386/BSD, a group of avid 386/BSD users formed the NetBSD group to pool their collective resources to help maintain and later enhance the system. By early 1993 they were doing releases that became known as the NetBSD distribution. The NetBSD group chose to emphasize the support of as many platforms as possible and continued the research-style development done by the CSRG. Until 1998, their distribution was done solely over the net with no distribution media available. Their group continues to target primarily the hard-core technical users.

The FreeBSD group was formed a few months after the NetBSD group with a charter to support just the PC architecture and to go after a larger and less technically advanced audience, much as Linux had done. They built elaborate installation scripts and began shipping their system on a low-cost CD-ROM in December 1993. The combination of ease of installation and heavy promotion on the net and at major trade shows, such as Comdex, led to a large, rapid growth curve. FreeBSD quickly rose to have the largest installed base of all the Networking Release 2-derived systems.

FreeBSD also rode the wave of Linux popularity by adding a Linux emulation mode that allows Linux binaries to run on the FreeBSD platform. This feature allows FreeBSD users to use the ever growing set of applications available for Linux while getting the robustness, reliability, and performance of the FreeBSD system.

In 1995, OpenBSD spun off from the NetBSD group. Their technical focus was aimed at improving the security of the system. Their marketing focus was to make the system easier to use and more widely available. Thus, they began producing and selling CD-ROMs, with many of the ease of installation ideas from the FreeBSD distribution.

The Lawsuit

In addition to the groups organized to freely redistribute systems originating from the Networking Release 2 tape, a company, Berkeley Software Design Incorporated (BSDI), was formed to develop and distribute a commercially supported version of the code. Like the other groups, it started by adding the six missing files that Bill Jolitz had written for his 386/BSD release. BSDI began selling its system, including both source and binaries, in January 1992 for $995. It began running advertisements touting its 99 percent discount over the price charged for System V source plus binary systems. Interested readers were told to call 1-800-ITS-UNIX.

Shortly after BSDI began its sales campaign, it received a letter from UNIX System Laboratory (USL) (a mostly owned subsidiary of AT&T spun off to develop and sell UNIX) [Ritchie, 2004]. The letter demanded that BSDI stop promoting its product as UNIX and in particular that it stop using the deceptive phone number. Although the phone number was promptly dropped and the advertisements changed to explain that the product was not UNIX, USL was still unhappy and filed suit to enjoin BSDI from selling its product. The suit alleged that the BSDI product contained USL proprietary code and trade secrets. USL sought to get an injunction to halt BSDI's sales until the lawsuit was resolved claiming that it would suffer irreparable harm from the loss of its trade secrets if the BSDI distributions continued.

At the preliminary hearing for the injunction, BSDI contended that it was simply using the sources being freely distributed by the University of California plus six additional files. BSDI was willing to discuss the content of any of the six added files but did not believe it should be held responsible for the files being distributed by the University of California. The judge agreed with BSDI's argument and told USL that it would have to restate its complaint based solely on the six files or the case would be dismissed. Recognizing that it would have a hard time making a case from just the six files, USL decided to refile the suit against both BSDI and the University of California. As before, USL requested an injunction on the shipping of Networking Release 2 from the university and of the BSDI products.

With the impending injunction hearing just a few short weeks away, preparation began in earnest. All the members of the CSRG were deposed, as was nearly everyone employed by BSDI. Briefs, counterbriefs, and counter-counterbriefs flew back and forth between the lawyers. The staff of the CSRG turned from writing code to writing several hundred pages of material that found its way into various briefs.

In December 1992, Dickinson R. Debevoise, a United States District Judge in New Jersey, heard the arguments for the injunction. Although judges usually rule on injunction requests immediately, he decided to take it under advisement. On a Friday about six weeks later, he issued a 40-page opinion in which he denied the injunction and threw out all but two of the complaints [Debevoise, 1993]. The remaining two complaints were narrowed to recent copyrights and the possibility of the loss of trade secrets. He also suggested that the matter should be heard in a state court system before being heard in federal court.

The University of California took the hint and rushed to California state court the following Monday morning with a countersuit against USL. By filing first in California, the university had established the locale of any further state court action. Constitutional law requires all state filing to be done in a single state to prevent a litigant with deep pockets from bleeding an opponent dry by filing 50 cases against them, one in each state. The result was that if USL wanted to take any action against the university in state courts, it would be forced to do so in California rather than in its home state of New Jersey.

The university's suit claimed that USL had failed in its obligation to provide due credit to the university for the use of BSD code in System V as required by the license that it had signed with the university [Linzner & MacDonald, 1993]. If the claim were found to be valid, the university asked that USL be forced to reprint all its documentation with the appropriate due credit added, to notify all its licensees of its oversight, and to run full-page advertisements in major publications such as the Wall Street Journal and Fortune magazine, notifying the business world of its inadvertent oversight.

Soon after the filing in state court, USL was bought from AT&T by Novell. The CEO of Novell, Ray Noorda, stated publicly that he would rather compete in the marketplace than in court. By the summer of 1993 settlement talks had started. Unfortunately, the two sides had dug in so deep that the talks proceeded very slowly. With some further prodding by Ray Noorda on the USL side, many of the sticking points were removed, and a settlement was finally reached in January 1994. The result was that three files were removed from the 18,000 that made up Networking Release 2, and a few minor changes were made to other files. In addition, the university agreed to add USL copyrights to about 70 files, although those files continued to be freely redistributed.

4.4BSD

The newly blessed release was called 4.4BSD-Lite and was released in June 1994 under terms identical to those used for the Networking releases. Specifically, the terms allow free redistribution in source and binary form, subject only to the constraint that the university copyrights remain intact and that the university receive credit when others use the code. Simultaneously, the complete system was released as 4.4BSD-Encumbered, which still required recipients to have a USL source license.

The lawsuit settlement also stipulated that USL would not sue any organization using 4.4BSD-Lite as the base for its system. So all the extant BSD groups— BSDI, NetBSD, and FreeBSD—had to restart their code base with the 4.4BSD-Lite sources into which they then merged their enhancements and improvements. While this reintegration caused a short-term delay in the development of the various BSD systems, it was a blessing in disguise, since it forced all the divergent groups to resynchronize with the three years of development that had occurred at the CSRG since the release of Networking Release 2.

4.4BSD-Lite Release 2

The money received from the 4.4BSD-Encumbered and 4.4BSD-Lite releases was used to fund a part-time effort to integrate bug fixes and enhancements. These changes continued for two years until the rate of bug reports and feature enhancements had died down to a trickle. The final set of changes was released as 4.4BSD-Lite Release 2 in June 1995. Most of the changes incorporated into 4.4BSD-Lite Release 2 eventually made it into the other systems' source bases.

Though the license term requiring that due credit be given to the university had been extremely helpful in the lawsuit, the university agreed to drop it following the final release. As many people began using the BSD-style copyrights for their own code, the proliferation of due-credit clauses in open-source software became difficult to determine and unmanageably large. By agreeing to drop the due-credit clause, the university hoped to set an example for others using its license. Over time, and with a lot of effort from the BSD community, the due-credit clause has been dropped from many of the open-source programs that use the BSD-style license.

Following the release of 4.4BSD-Lite Release 2, the CSRG was disbanded. After 15 years of piloting the BSD ship, it was time to let others with fresh ideas and boundless enthusiasm take over. While it might seem best to have a single centralized authority overseeing the system development, the idea of having several groups with different charters ensures that many different approaches will be tried and that there is no single point of failure. Because the system is released in source form, the best ideas can easily be picked up by other groups. Indeed, cross-pollination of ideas between open-source projects is common.


1.4. The FreeBSD Development Model

Running an open-source project is different from running traditional software development. In traditional development, the staff are paid, so it is possible to have managers and a system architect that set schedules and direct the programmers' activities. With open source, the developers are volunteers. They tend to be transient, usually doing a project or two before finding some other activity on which they prefer to spend their free time. They cannot be directed because they only work on what interests them. Because their jobs, families, and social lives often take precedence over their work on the project, it is impossible to put together schedules. Finally, there is no paid staff to fill the management role of traditional development. Thus, a successful open-source-development project must be self-organizing and set up to gracefully handle a high turnover of its active developers.

The development model used by FreeBSD (as well as NetBSD and OpenBSD) was first set in motion by the CSRG [McKusick et al., 1989]. The CSRG was always a small group of software developers. This resource limitation required careful software-engineering management. Careful coordination was needed not only of the CSRG personnel but also of members of the general community who contributed to the development of the system. Certain outside developers had permission to modify the master copy of the system source directly. People given access to the master sources were carefully screened beforehand but were not closely supervised. Everyone committing changes to the system source received notification of all changes, allowing everyone to be aware of changes going into the system. Everyone was required to have any nontrivial changes reviewed by at least one other person before committing them to the tree. This model allowed many lines of development to proceed concurrently while still keeping the project coherent.

The FreeBSD project is organized in much the same way as the CSRG. The entire FreeBSD project, including all the source code, documentation, bug reports, mailing-list archives, and even administrative data, is maintained in a publicly readable source-code-control system. Anyone may view the source code and existing bug reports, track progress on fixing bugs, and post bug reports. Anyone may join and participate in the numerous FreeBSD mailing lists. There are three groups of people that directly work on FreeBSD: developers, committers, and the core team.

There are 3000 to 4000 developers, each of whom works on some part of the system such as maintaining the FreeBSD kernel, continuing development of the 1000 core FreeBSD utilities, writing FreeBSD documentation, and updating other open-source software in the FreeBSD ports collection. Developers are able to access the source-code repository, but they are not permitted to change it. Instead they must work with a committer or file a problem report to get their changes added to the system.

There are currently 300 to 400 committers. Like the developers, most of them specialize in some part of the system. Unlike the developers, they are permitted to make changes to those parts of the source-code repository in which they have been authorized to work. All nontrivial changes should be reviewed by one or more other committers before being checked into the source tree. Most committers are doing work of their own as well as reviewing and committing the work of several developers.

Nomination for advancement from developer to committer is done by the existing committers. Most commonly a developer will be nominated by the committer with whom he has been working. The nomination, along with a description and evaluation of past work and an initial scope of new work, is sent to the core team for approval.

At the center of the project is the core team. The core team is composed of nine people who are elected every two years. The candidates for the core team come from the committers and the committers elect the core team. The core team acts as the final gatekeepers of the source code. They monitor what is being committed and resolve conflicts if two or more committers cannot agree on how to solve a particular problem. The core team also approves developers to be advanced to being committers and in (rare circumstances) temporarily or permanently evicts someone from the committer group. The usual reason for departure from the committer group is inactivity (making no changes to the system for more than a year).

The development structure of the FreeBSD project is directly derived from the one that we had established at the CSRG. Both the CSRG and FreeBSD use a central source-code-controlled repository. The FreeBSD core team is analogous to the CSRG staff. The FreeBSD committers are much like the people to whom Berkeley gave accounts on the CSRG development machine that allowed them to commit changes to the CSRG sources. And the FreeBSD developers are similar to the people that contributed to Berkeley, but they did not have accounts on the CSRG development machine.

The FreeBSD project has made some important improvements. First, they recognize that even the most dedicated programmer will eventually burn out, lose interest, or otherwise decide to move on. There must be some way to let these people gracefully step aside rather than letting their inattention create a void at a critical point in the project. So unlike the CSRG model of having staff that were dictators for life, FreeBSD went to an elected core that is answerable to the committers. A core member who is burned out can decide (or be persuaded) not to run for reelection when his or her term ends. Core members who are not serving the interest of the committers will not be reelected. Equally important, active and energetic people have plenty of opportunity to move up through the ranks. Because the core team is elected, people rise into that rank because their peers actively working on the project feel that they should have the job. This approach works better than advancing because you are good buddies with somebody at the top. It also ensures that the core team is made up of those who are good at communicating with others, an important skill to have in that position.

Another significant improvement made by the FreeBSD project is to automate many tasks and set up remote mirrors of the source-code repository, Web site, and bug reports. These changes have allowed the project to support many more contributors than would have been possible under the CSRG model. The FreeBSD project has also managed to become much less United States-centric by welcoming developers from around the world, including active people in Japan, Australia, Russia, South Africa, Denmark, France, Germany, and the United Kingdom, to name just a few of the countries with active FreeBSD development.

The CSRG used to release new versions of the system about every two years. Changes to these distributions were rare, typically only small and critical security-or stability-related changes. Between versions, the CSRG would do test releases to gain experience with the new features that were being developed.

The FreeBSD project has greatly expanded on the CSRG distribution scheme. At any point in time there are two FreeBSD distributions. The first is the "stable" release that is intended to be used in production environments. The second is the "current" release that represents the current state of the FreeBSD system and is intended for use by developers and users needing the latest features.

The stable release changes slowly, and the changes are limited to fixing bugs, improving performance, and adding incremental hardware support. The stable system is released three to four times per year, although users wishing to upgrade more often can download and install the latest stable code as frequently as they need to do so (for example, after a major security patch has been made). The stable version of FreeBSD is analogous to the CSRG major-version releases except that they are more actively updated and are made available to the users. Like the stable release, snapshots of the current release are created every few months. However, most users of the current release update much more frequently (daily updates are common). By having mirrored copies of the stable and current distributions available throughout the world, the FreeBSD project allows its worldwide user base to stay up to date much more easily than was possible with the CSRG distributions.

About every two years, the current branch is forked to create a new stable release. Once the new stable branch has proven to be reliable enough for production use, work largely ceases on the old stable branch and production users switch over to the new stable release. The mainline development continues on the current branch. Nearly all changes are made first to the current branch. Only after a change has been tested in the current branch and proven to work in that environment is it merged-from-current (MFC-ed) to the stable release.

One advantage that the CSRG long had over the FreeBSD project was that the CSRG was part of the University of California at Berkeley. Since the university is a nonprofit organization, contributions made to the CSRG were tax-deductible to the contributor. Some people at the FreeBSD project had long felt that they should find a way to let contributors to the project get a tax deduction. A few years ago, they set up the FreeBSD Foundation, which after three years of good nonprofit work, was granted 501(c)3 status by the United States taxing authorities. This certification means that contributions made to the FreeBSD Foundation can be deducted from United States federal and state taxes in the same way as a contribution made to the university can be deducted. The ability to get a tax deduction has markedly increased the volume of monetary contributions to the FreeBSD project, which has enabled them to fund development of parts of the system that are tedious to create but necessary and important.

Over the past 10 years the FreeBSD project has grown at a steady but sustainable pace. Although Linux has attracted a mass following, FreeBSD continues to hold its place in the high-performance-server space. Indeed, Linux has helped to evangelize the viability of open source to the corporate marketplace, and FreeBSD has ridden on its coattails. It is far easier to convince your management to switch from Linux to FreeBSD than it is to convince them to move from Microsoft's Windows to FreeBSD. Linux has also supplied a steady stream of developers for FreeBSD. Until recently Linux had no central source-code repository, so to contribute you had to work for a Linux distributor or you had to get the ear of a member of the small set of people who could get changes put into the system. The much more egalitarian and merit-based organization of the FreeBSD project has provided a steady influx of high-quality developers. The typical new committer to the FreeBSD project is in their mid- to late 20s and has been programming Linux or other open-source projects for a decade. These people have enough experience and maturity that they are quickly able to become effective contributors to the project. And the mentoring inherent in the progression of developer to committer ensures that by the time someone has the right to directly commit code to the FreeBSD tree, they understand the style and code-clarity guidelines that are critically important to preserving the quality, robustness, and maintainability of FreeBSD.

The goal of the FreeBSD project is to provide software that may be used for any purpose and without strings attached. Many of the developers have a significant investment in the code (and project) and certainly do not mind a little financial compensation occasionally, but they certainly do not insist on it. They believe that their first and foremost mission is to provide code to any and all comers, for whatever purpose, so that the code gets the widest possible use and provides the greatest possible benefit [Hubbard, 2004].


References

AT&T, 1987. AT&T, The System V Interface Definition (SVID), Issue 2, American Telephone and Telegraph, Murray Hill, NJ, January 1987.

Babao [image] lu &Joy, 1981. Ö. Babaolu & W. N. Joy, "Converting a Swap-Based System to Do Paging in an Architecture Lacking Page-Referenced Bits," Proceedings of the Eighth Symposium on Operating Systems Principles, pp. 78-86, December 1981.

Bach, 1986. M. J. Bach, The Design of the UNIX Operating System, Prentice-Hall, Englewood Cliffs, NJ, 1986.

Comer, 2000. D. Comer, Internetworking with TCP/IP Volume 1, 4th ed., Prentice-Hall, Upper Saddle River, NJ, 2000.

Comer, 1984. D. Comer, Operating System Design: The Xinu Approach, Prentice-Hall, Englewood Cliffs, NJ, 1984.

Compton, 1985. M. Compton, editor, "The Evolution of UNIX," UNIX Review, vol. 3, no. 1, January 1985.

Debevoise, 1993. D. Debevoise, Civ. No. 92-1667, Unix System Laboratories Inc. vs. Berkeley Software Design Inc., http://sco.tuxrocks.com/Docs/USL/Doc-92.html, March 3, 1993.

DiBona et al., 1999. C. DiBona, S. Ockman, & M. Stone, Open Sources: Voices from the Open Source Revolution, pp. 31-46, Chapter 2—Twenty Years of Berkeley Unix: From AT&T-Owned to Freely Redistributable, http://www.oreilly.com/catalog/opensources/book/kirkmck.html, ISBN 1-56592-582-3, O'Reilly & Associates, Inc., Sebastopol, CA 95472, 1999.

Ewensetal., 1985. P Ewens, D. R. Blythe, M. Funkenhauser, & R. C. Holt, "Tunis: A Distributed Multiprocessor Operating System," USENIX Association Conference Proceedings, pp. 247-254, June 1985.

Holt, 1983. R. C. Holt, Concurrent Euclid, the UNIX System, and Tunis, Addison-Wesley, Reading, MA, 1983.

Hubbard, 2004. J. Hubbard, "A Brief History of FreeBSD," FreeBSD Handbook, section 1.3.1, http://www.freebsd.org/doc/en_US.IS08859-1/books/handbook/history.html, March 2004.

ISO, 1999. ISO, "ISO/IEC 9899 Programming Language C Standard," ISO 9899, can be orderd from http://www.iso.org, December, 1999.

Joy, 1980. W. N. Joy, "Comments on the Performance of UNIX on the VAX," Technical Report, University of California Computer System Research Group, Berkeley, CA, April 1980.

Jung, 1985. R. S. Jung, "Porting the AT&T Demand Paged UNIX Implementation to Microcomputers," USENIX Association Conference Proceedings, pp. 361-370, June 1985.

Kashtan, 1980. D. L. Kashtan, "UNIX and VMS: Some Performance Comparisons," Technical Report, SRI International, Menlo Park, CA, February 1980.

Kernighan & Ritchie, 1978. B. W. Kernighan & D. M. Ritchie, The C Programming Language, Prentice-Hall, Englewood Cliffs, NJ, 1978.

Kernighan & Ritchie, 1989. B. W. Kernighan & D. M. Ritchie, The C Programming Language, 2nd ed., Prentice-Hall, Englewood Cliffs, NJ, 1989.

Linzner & MacDonald, 1993. J. Linzner & M. MacDonald, University of California at Berkeley versus Unix System Laboratories Inc., http://cm.bell-labs.com/cm/cs/who/dmr/bsdi/930610.ucb_complaint.txt, June 1993.

McKusick et al., 1989. M. K. McKusick, M. Karels, & K. Bostic, "The Release Engineering of 4.3BSD," Proceedings of the New Orleans Usenix Workshop on Software Management, pp. 95-100, April 1989.

Miller, 1978. R. Miller, "UNIX—A Portable Operating System," ACM Operating System Review, vol. 12, no. 3, pp. 32-37, July 1978.

Miller, 1984. R. Miller, "A Demand Paging Virtual Memory Manager for System V," USENIXAssociation Conference Proceedings, pp. 178-182, June 1984.

Mohr, 1985. A. Mohr, "The Genesis Story," UNIX Review, vol. 3, no. 1, p. 18, January 1985.

Organick, 1975. E. I. Organick, The Multics System: An Examination of Its Structure, MIT Press, Cambridge, MA, 1975.

P1003.1, 1988. P1003.1, IEEE P1003.1 Portable Operating System Interface for Computer Environments (POSIX), Institute of Electrical and Electronic Engineers, Piscataway, NJ, 1988.

Peirce, 1985. N. Peirce, "Putting UNIX in Perspective: An Interview with Victor Vyssotsky," UNIX Review, vol. 3, no. 1, p. 58, January 1985.

Presotto & Ritchie, 1985. D. L. Presotto & D. M. Ritchie, "Interprocess Communication in the Eighth Edition UNIX System," USENIX Association Conference Proceedings, pp. 309-316, June 1985.

Richards & Whitby-Strevens, 1980. M. Richards & C. Whitby-Strevens, BCPL: The Language and Its Compiler, Cambridge University Press, Cambridge, U.K., 1980, 1982.

Ritchie, 1978. D. M. Ritchie, "A Retrospective," Bell System Technical Journal, vol. 57, no. 6, pp. 1947-1969, July-August 1978.

Ritchie, 1984a. D. M. Ritchie, "The Evolution of the UNIX Time-Sharing System," AT&T Bell Laboratories Technical Journal, vol. 63, no. 8, pp. 1577-1593, October 1984.

Ritchie, 1987. D. M. Ritchie, "Unix: A Dialectic," USENIX Association Conference Proceedings, pp. 29-34, January 1987.

Ritchie, 1984b. D. M. Ritchie, "Reflections on Software Research," Comm ACM, vol. 27, no. 8, pp. 758-760, 1984.

Ritchie, 2004. D. M. Ritchie, Documents on Unix System Laboratories Inc. versus Berkeley Software Design Inc., http://cm.bell-labs.com/cm/cs/who/dmr/bsdi/bsdisuit.html, March 2004.

Ritchie et al., 1978. D. M. Ritchie, S. C. Johnson, M. E. Lesk, & B. W. Kernighan, "The C Programming Language," Bell System Technical Journal, vol. 57, no. 6, pp. 1991-2019, July-August 1978.

Rosler, 1984. L. Rosler, "The Evolution of C—Past and Future," AT&T Bell Laboratories Technical Journal, vol. 63, no. 8, pp. 1685-1699, October 1984.

Tanenbaum, 1987. A. S. Tanenbaum, Operating Systems: Design and Implementation, Prentice-Hall, Englewood Cliffs, NJ, 1987.

Tuthill, 1985. B. Tuthill, "The Evolution of C: Heresy and Prophecy," UNIX Review, vol. 3, no. 1, p. 80, January 1985.

Wilson, 1985. O. Wilson, "The Business Evolution of the UNIX System," UNIX Review, vol. 3, no. 1, p. 46, January 1985.

X/OPEN, 1987. X/OPEN, The X/OPEN Portability Guide (XPG), Issue 2, Elsevier Science, Amsterdam, Netherlands, 1987.


Chapter 2. Design Overview of FreeBSD


2.1. FreeBSD Facilities and the Kernel

The FreeBSD kernel provides four basic facilities: processes, a filesystem, communications, and system startup. This section outlines where each of these four basic services is described in this book.

  1. A process is composed of an address space with one or more threads of control running within it. Mechanisms for creating, terminating, and otherwise controlling processes are discussed in Chapter 4. The system multiplexes separate virtual-address spaces for each process; this memory management is discussed in Chapter 5.

  2. The user interface to the filesystem and devices is similar; common aspects are discussed in Chapter 6. The organization and management of the devices in the I/O subsystem is described in Chapter 7. The filesystem provides operations to manipulate a set of named files, organized in a tree-structured hierarchy of directories. The filesystem must organize the storage of these files and directories on physical media such as disks. The role of the filesystem in doing these tasks is presented in Chapter 8. Access to files on remote machines is the subject of Chapter 9. Pseudo-terminals are used to access the system; their operation is the subject of Chapter 10.

  3. Communication mechanisms provided by traditional UNIX systems include simplex reliable byte streams between related processes (see pipes, Section 11.1), and notification of exceptional events (see signals, Section 4.7). FreeBSD also has a general interprocess-communication facility. This facility, described in Chapter 11, uses access mechanisms distinct from those of the filesystem, but once a connection is set up, a process can access it as though it were a pipe. There is a general networking framework, discussed in Chapter 12, that is normally used as a layer underlying the IPC facility. Chapter 13 describes a particular networking implementation in detail.

  4. Any real operating system has operational issues, such as how to start it running. Startup and operational issues are described in Chapter 14.

Sections 2.3 through 2.14 present introductory material related to Chapters 3 through 14. We define terms, examine basic system calls, and explore historical developments. Finally, we give the reasons for many major design decisions.

The Kernel

The kernel is the part of the system that runs in protected mode and mediates access by all user programs to the underlying hardware (e.g., CPU, keyboard, monitor, disks, network links) and software constructs (e.g., filesystem, network protocols). The kernel provides the basic system facilities; it creates and manages processes and provides functions to access the filesystem and communication facilities. These functions, called system calls, appear to user processes as library subroutines. These system calls are the only interface that processes have to these facilities. Details of the system-call mechanism are given in Chapter 3, as are descriptions of several kernel mechanisms that do not execute as the direct result of a process doing a system call.

A kernel, in traditional operating-system terminology, is a small nucleus of software that provides only the minimal facilities necessary for implementing additional operating-system services. Through much of the 1980s, research operating systems—such as Chorus [Rozier et al., 1988], Mach [Accetta et al., 1986], Tunis [Ewens et al., 1985], and the V Kernel [Cheriton, 1988]—attempted to make this division of functionality into more than just a logical one. Services such as filesystems and networking protocols were implemented as client application processes of the nucleus or kernel. These micro-kernels largely failed because of the high overhead of transitioning between kernel processes.

The FreeBSD kernel is not partitioned into multiple processes. This basic design decision was made in the earliest versions of UNIX. The first two implementations by Ken Thompson had no memory mapping and thus made no hardware-enforced distinction between user and kernel space [Ritchie, 1988]. A message-passing system could have been implemented as readily as the actually implemented model of kernel and user processes. The monolithic kernel was chosen for simplicity and performance. And the early kernels were small; the inclusion of facilities such as networking into the kernel has increased its size, although the kernel is still small compared to many of the applications that run on it.

Users ordinarily interact with the system through a command-language interpreter, called a shell, and through additional user application programs. Such programs and the shell are implemented with processes rather than being part of the kernel. Details of such programs are beyond the scope of this book, which instead concentrates almost exclusively on the kernel.

Sections 2.3 and 2.4 describe the services provided by the FreeBSD kernel and give an overview of the latter's design. Later chapters describe the detailed design and implementation of these services as they appear in FreeBSD.


2.2. Kernel Organization

In this section, we view the organization of the FreeBSD kernel in two ways.

  1. As a static body of software, categorized by the functionality offered by the modules that make up the kernel

  2. By its dynamic operation, categorized according to the services provided to users

The largest part of the kernel implements the system services that applications access through system calls. In FreeBSD, this software has been organized according to the following:

  • Basic kernel facilities: timer and system-clock handling, descriptor management, and process management

  • Memory-management support: paging and swapping

  • Generic system interfaces: the I/O, control, and multiplexing operations performed on descriptors

  • The filesystem: files, directories, pathname translation, file locking, and I/O buffer management

  • Terminal-handling support: the pseudo-terminal interface and terminal line disciplines

  • Interprocess-communication facilities: sockets

  • Support for network communication: communication protocols and generic network facilities, such as routing

Most of the software in these categories is machine independent and is portable across different hardware architectures.

The machine-dependent aspects of the kernel are isolated from the mainstream code. In particular, none of the machine-independent code contains conditional code for specific architectures. When an architecture-dependent action is needed, the machine-independent code calls an architecture-dependent function that is located in the machine-dependent code. The software that is machine dependent includes the following.

  • Low-level system-startup actions

  • Trap and fault handling

  • Low-level manipulation of the run-time context of a process

  • Configuration and initialization of hardware devices

  • Run-time support for I/O devices

Table 2.1 summarizes the machine-independent software that constitutes the FreeBSD kernel for the PC. The numbers in column 2 are for lines of C source code, header files, and assembly language. Virtually all the software in the kernel is written in the C programming language; a mere 0.6 percent is written in assembly language. As the statistics in Table 2.2 show, the machine-dependent software, excluding device support, accounts for a minuscule 6.9 percent of the kernel. Not shown are the 846,525 lines of code for the hundreds of supported devices, only a few of which will be loaded into any particular kernel.

Table 2.1. Machine-independent software in the FreeBSD kernel.

Category

Lines of Code

Percentage of Kernel

headers

38,158

4.8%

initialization

1,663

0.2%

kernel facilities

53,805

6.7%

generic interfaces

22,191

2.8%

interprocess communication

10,019

1.3%

terminal handling

5,798

0.7%

virtual memory

24,714

3.1%

vnode management

22,764

2.9%

local filesystem

28,067

3.5%

miscellaneous filesystems (19)

58,753

7.4%

network filesystem

22,436

2.8%

network communication

46,570

5.8%

Internet V4 protocols

41,220

5.2%

Internet V6 protocols

45,527

5.7%

IPsec

17,956

2.2%

netgraph

74,338

9.3%

cryptographic support

7,515

0.9%

GEOM layer

11,563

1.4%

CAM layer

41,805

5.2%

ATA layer

14,192

1.8%

ISA bus

10,984

1.4%

PCI bus

72,366

9.1%

pccard bus

6,916

0.9%

Linux compatibility

10,474

1.3%

total machine independent

689,794

86.4%

Key: GEOM—geometry; CAM—Common Access Method; ATA—Advanced Technology Attachment; ISA—Industry Standard Architecture; PCI—Peripheral Component Interconnect.

Table 2.2. Machine-dependent software for the PC in the FreeBSD kernel.

Category

Lines of Code

Percentage of Kernel

machine dependent headers

16,115

2.0%

ISA bus

50,882

6.4%

PCI bus

2,266

0.3%

virtual memory

3,118

0.4%

other machine dependent

26,708

3.3%

routines in assembly language

4,400

0.6%

Linux compatibility

4,857

0.6%

Total machine dependent

108,346

13.6%

Key: ISA—Industry Standard Architecture; PCI—Peripheral Component Interconnect.

Only a small part of the kernel is devoted to initializing the system. This code is used when the system is bootstrapped into operation and is responsible for setting up the kernel hardware and software environment (see Chapter 14). Some operating systems (especially those with limited physical memory) discard or overlay the software that performs these functions after that software has been executed. The FreeBSD kernel does not reclaim the memory used by the startup code because that memory space is barely 0.2 percent of the kernel resources used on a typical machine. Also, the startup code does not appear in one place in the kernel—it is scattered throughout, and it usually appears in places logically associated with what is being initialized.


2.3. Kernel Services

The boundary between the kernel- and user-level code is enforced by hardware-protection facilities provided by the underlying hardware. The kernel operates in a separate address space that is inaccessible to user processes. Privileged operations-such as starting I/O and halting the central processing unit (CPU)—are available to only the kernel. Applications request services from the kernel with system calls. System calls are used to cause the kernel to execute complicated operations, such as writing data to secondary storage, and simple operations, such as returning the current time of day. All system calls appear synchronous to applications: The application does not run while the kernel does the actions associated with a system call. The kernel may finish some operations associated with a system call after it has returned. For example, a write system call will copy the data to be written from the user process to a kernel buffer while the process waits, but it will usually return from the system call before the kernel buffer is written to the disk.

A system call usually is implemented as a hardware trap that changes the CPU's execution mode and the current address-space mapping. Parameters supplied by users in system calls are validated by the kernel before being used. Such checking ensures the integrity of the system. All parameters passed into the kernel are copied into the kernel's address space to ensure that validated parameters are not changed as a side effect of the system call. System-call results are returned by the kernel, either in hardware registers or by their values being copied to user-specified memory addresses. Like parameters passed into the kernel, addresses used for the return of results must be validated to ensure that they are part of an application's address space. If the kernel encounters an error while processing a system call, it returns an error code to the user. For the C programming language, this error code is stored in the global variable errno, and the function that executed the system call returns the value -1.

User applications and the kernel operate independently of each other. FreeBSD does not store I/O control blocks or other operating-system-related data structures in the application's address space. Each user-level application is provided an independent address space in which it executes. The kernel makes most state changes, such as suspending a process while another is running, invisible to the processes involved.


2.4. Process Management

FreeBSD supports a multitasking environment. Each task or thread of execution is termed a process. The context of a FreeBSD process consists of user-level state, including the contents of its address space and the run-time environment, and kernel-level state, which includes scheduling parameters, resource controls, and identification information. The context includes everything used by the kernel in providing services for the process. Users can create processes, control the processes' execution, and receive notification when the processes' execution status changes. Every process is assigned a unique value, termed a process identifier (PID). This value is used by the kernel to identify a process when reporting status changes to a user, and by a user when referencing a process in a system call.

The kernel creates a process by duplicating the context of another process. The new process is termed a child process of the original parent process. The context duplicated in process creation includes both the user-level execution state of the process and the process's system state managed by the kernel. Important components of the kernel state are described in Chapter 4.

The process lifecycle is depicted in Figure 2.1. A process may create a new process that is a copy of the original by using the fork system call. The fork call returns twice: once in the parent process, where the return value is the process identifier of the child, and once in the child process, where the return value is 0. The parent-child relationship induces a hierarchical structure on the set of processes in the system. The new process shares all its parent's resources, such as file descriptors, signal-handling status, and memory layout.

Figure 2.1. Process-management system calls.

[View full size image]

Although there are occasions when the new process is intended to be a copy of the parent, the loading and execution of a different program is a more useful and typical action. A process can overlay itself with the memory image of another program, passing to the newly created image a set of parameters, using the system call execve. One parameter is the name of a file whose contents are in a format recognized by the system—either a binary-executable file or a file that causes the execution of a specified interpreter program to interpret its contents.

A process may terminate by executing an exit system call, sending 8 bits of exit status to its parent. If a process wants to communicate more than a single byte of information with its parent, it must either set up an interprocess-communication channel using pipes or sockets, or use an intermediate file. Interprocess communication is discussed extensively in Chapter 11.

A process can suspend execution until any of its child processes terminate using the wait system call, which returns the PID and exit status of the terminated child process. A parent process can arrange to be notified by a signal when a child process exits or terminates abnormally. Using the wait4 system call, the parent can retrieve information about the event that caused termination of the child process and about resources consumed by the process during its lifetime. If a process is orphaned because its parent exits before it is finished, then the kernel arranges for the child's exit status to be passed back to a special system process (init: see Sections 3.1 and 14.5). The details of how the kernel creates and destroys processes are given in Chapter 5.

Processes are scheduled for execution according to a process-priority parameter. Under the default timesharing scheduler, this priority is managed by a kernel-based scheduling algorithm. Users can influence the scheduling of a process by specifying a parameter (nice) that weights the overall scheduling priority but are still obligated to share the underlying CPU resources according to the kernel's scheduling policy. FreeBSD also has a real-time scheduler. Processes running under the real-time scheduler manage their own priority, which is not changed by the kernel. The kernel will run the highest priority real-time process to the exclusion of all other processes. Thus, real-time processes are not obliged to share the underlying CPU resources.

Signals

The system defines a set of signals that may be delivered to a process. Signals in FreeBSD are modeled after hardware interrupts. A process may specify a user-level subroutine to be a handler to which a signal should be delivered. When a signal is generated, it is blocked from further occurrence while it is being caught by the handler. Catching a signal involves saving the current process context and building a new one in which to run the handler. The signal is then delivered to the handler, which can either abort the process or return to the executing process (perhaps after setting a global variable). If the handler returns, the signal is unblocked and can be generated (and caught) again.

Alternatively, a process may specify that a signal is to be ignored or that a default action, as determined by the kernel, is to be taken. The default action of certain signals is to terminate the process. This termination may be accompanied by creation of a core file that contains the current memory image of the process for use in postmortem debugging.

Some signals cannot be caught or ignored. These signals include SIGKILL, which kills runaway processes, and the job-control signal SIGSTOP.

A process may choose to have signals delivered on a special stack so that sophisticated software stack manipulations are possible. For example, a language supporting coroutines needs to provide a stack for each coroutine. The language run-time system can allocate these stacks by dividing up the single stack provided by FreeBSD. If the kernel does not support a separate signal stack, the space allocated for each coroutine must be expanded by the amount of space required to catch a signal.

All signals have the same priority. If multiple signals are pending simultaneously, the order in which signals are delivered to a process is implementation specific. Signal handlers execute with the signal that caused their invocation to be blocked, but other signals may yet occur. Mechanisms are provided so that processes can protect critical sections of code against the occurrence of specified signals.

The design and implementation of signals are described in Section 4.7.

Process Groups and Sessions

Processes are organized into process groups. Process groups are used to control access to terminals and to provide a means of distributing signals to collections of related processes. A process inherits its process group from its parent process. Mechanisms are provided by the kernel to allow a process to alter its process group or the process group of its descendants. Creating a new process group is easy; the value of a new process group is ordinarily the process identifier of the creating process.

The group of processes in a process group is sometimes referred to as a job and is manipulated by high-level system software, such as the shell. A common kind of job created by a shell is a pipeline of several processes connected by pipes, such that the output of the first process is the input of the second, the output of the second is the input of the third, and so forth. The shell creates such a job by forking a process for each stage of the pipeline, and then putting all those processes into a separate process group.

A user process can send a signal to each process in a process group as well as to a single process. A process in a specific process group may receive software interrupts affecting the group, causing the group to suspend or resume execution, or to be interrupted or terminated.

A terminal (or more commonly a software emulation of a terminal called a pseudo-terminal) has a process-group identifier assigned to it. This identifier is normally set to the identifier of a process group associated with the terminal. A job-control shell may create several process groups associated with the same terminal; the terminal is the controlling terminal for each process in these groups. A process may read from a descriptor for its controlling terminal only if the terminal's process-group identifier matches that of the process. If the identifiers do not match, the process will be blocked if it attempts to read from the terminal. By changing the process-group identifier of the terminal, a shell can arbitrate a terminal among several different jobs. This arbitration is called job control and is described, with process groups, in Section 4.8.

Just as a set of related processes can be collected into a process group, a set of process groups can be collected into a session. The main uses for sessions are to create an isolated environment for a daemon process and its children, and to collect together a user's login shell and the jobs that that shell spawns.


2.5. Memory Management

Each process has its own private address space. The address space is initially divided into three logical segments: text, data, and stack. The text segment is read-only and contains the machine instructions of a program. The data and stack segments are both readable and writable. The data segment contains the initialized and uninitialized data portions of a program, whereas the stack segment holds the application's run-time stack. The stack segment is extended automatically by the kernel as the process executes. A process can expand or contract its data segment by making a system call, whereas a process can change the size of its text segment only when the segment's contents are overlaid with data from the filesystem or when debugging takes place. The initial contents of the segments of a child process are duplicates of the segments of a parent process.

The entire contents of a process address space do not need to be resident for a process to execute. If a process references a part of its address space that is not resident in main memory, the system pages the necessary information into memory. When system resources are scarce, the system uses a two-level approach to maintain available resources. If a modest amount of memory is available, the system will take memory resources away from processes if these resources have not been used recently. Should there be a severe resource shortage, the system will resort to swapping the entire context of a process to secondary storage. The demand paging and swapping done by the system are effectively transparent to processes. A process may, however, advise the system about expected future memory utilization as a performance aid.

BSD Memory-Management Design Decisions

The support of large sparse address spaces, mapped files, and shared memory was a requirement for 4.2BSD. An interface was specified, called mmap(), that allowed unrelated processes to request a shared mapping of a file into their address spaces. If multiple processes mapped the same file into their address spaces, changes to the file's portion of an address space by one process would be reflected in the area mapped by the other processes, as well as in the file itself. Ultimately, 4.2BSD was shipped without the mmap() interface, because of pressure to make other features, such as networking, available.

Further development of the mmap() interface continued during the work on 4.3BSD. Over 40 companies and research groups participated in the discussions leading to the revised architecture that was described in the Berkeley Software Architecture Manual [McKusick et al., 1994]. The first UNIX implementation of the interface was done by Sun Microsystems [Gingell et al., 1987].

Once again, time pressure prevented 4.3BSD from providing an implementation of the interface. Although the latter could have been built into the existing 4.3BSD virtual-memory system, the developers decided not to put it in because that implementation was nearly 10 years old. Furthermore, the original virtual-memory design was based on the assumption that computer memories were small and expensive, whereas disks were locally connected, fast, large, and inexpensive. Thus, the virtual-memory system was designed to be frugal with its use of memory at the expense of generating extra disk traffic. In addition, the 4.3BSD implementation was riddled with VAX memory-management hardware dependencies that impeded its portability to other computer architectures. Finally, the virtual-memory system was not designed to support the tightly coupled multiprocessors that are becoming increasingly common and important today.

Attempts to improve the old implementation incrementally seemed doomed to failure. A completely new design, on the other hand, could take advantage of large memories, conserve disk transfers, and have the potential to run on multiprocessors. Consequently, the virtual-memory system was completely replaced in 4.4BSD. The 4.4BSD virtual-memory system was based on the Mach 2.0 virtual-memory system [Tevanian, 1987], with updates from Mach 2.5 and Mach 3.0.

The FreeBSD virtual-memory system is an extensively tuned version of the virtual-memory implementation in 4.4BSD. It features efficient support for sharing, a clean separation of machine-independent and machine-dependent features, as well as multiprocessor support. Processes can map files anywhere in their address space. They can share parts of their address space by doing a shared mapping of the same file. Changes made by one process are visible in the address space of the other process and also are written back to the file itself. Processes can also request private mappings of a file, which prevents any changes that they make from being visible to other processes mapping the file or being written back to the file itself.

Another issue with the virtual-memory system is the way that information is passed into the kernel when a read or write system call is made. For these system calls, FreeBSD always copies data from the process address space into a buffer in the kernel. The copy is done for several reasons.

  • Often, the user data are not page aligned and are not a multiple of the hardware page length.

  • If the page is taken away from the process, it will no longer be able to reference that page. Some programs depend on the data remaining in the buffer even after those data have been written.

  • If the process is allowed to keep a copy of the page (as it is in current FreeBSD semantics), the page must be made copy-on-write. A copy-on-write page is one that is protected against being written by being made read-only. If the process attempts to modify the page, the kernel gets a write fault. The kernel then makes a copy of the page that the process can modify. Unfortunately, the typical process will immediately try to write new data to its output buffer, forcing the data to be copied anyway.

  • When pages are remapped to new virtual-memory addresses, most memory-management hardware requires that the hardware address-translation cache be purged selectively. The cache purges are often slow. The net effect is that remapping is slower than copying for blocks of data less than 4 to 8 Kbyte.

For read or write operations that are transferring large quantities of data, doing the copy can be time-consuming. An alternative to doing the copying is to remap the process memory into the kernel. The biggest incentives for memory mapping are the needs for accessing big files and for passing large quantities of data between processes. The mmap() interface provides a way for both of these tasks to be done without copying.

The mmap system call is not supported between processes running on different machines. Such processes must communicate using sockets connected across the network. Thus, sending the contents of a file across the network is another common operation where it is desirable to avoid copying. Historically, the sending of a file was done by reading the file into an application buffer, and then writing that buffer to a socket. This approach required two copies of the data: first from the kernel to the application buffer, and then from the application buffer back into the kernel to send on the socket. FreeBSD has the sendfile system call that sends data from a file down a socket without doing any copying.

Memory Management Inside the Kernel

The kernel often does allocations of memory that are needed for only the duration of a single system call. In a user process, such short-term memory would be allocated on the run-time stack. Because the kernel has a limited run-time stack, it is not feasible to allocate even moderate-sized blocks of memory on it. Consequently, such memory must be allocated through a more dynamic mechanism. For example, when the system must translate a pathname, it must allocate a 1-Kbyte buffer to hold the name. Other blocks of memory must be more persistent than a single system call, and thus could not be allocated on the stack even if there was space. An example is protocol-control blocks that remain throughout the duration of a network connection.

Demands for dynamic memory allocation in the kernel have increased as more services have been added. A generalized memory allocator reduces the complexity of writing code inside the kernel. Thus, the FreeBSD kernel has a general memory allocator that can be used by any part of the system. It has an interface similar to the C library routines malloc() and free() that provide memory allocation to application programs [McKusick & Karels, 1988]. Like the C library interface, the allocation routine takes a parameter specifying the size of memory that is needed. The range of sizes for memory requests is not constrained; however, physical memory is allocated and is not paged. The free routine takes a pointer to the storage being freed, but it does not require the size of the piece of memory being freed.

Some large and persistent allocations, such as the structure that tracks information about a process during its lifetime, are not well handled by the general memory allocator. The kernel provides a zone allocator for these types of allocations. Each memory type is given its own zone from which all its allocations are made. Memory allocated in one zone cannot be used by any other zone or by the general memory allocator. The semantics of the interface are similar to the general-memory allocator; memory is allocated from a zone with the zalloc() routine and freed with the zfree() routine.


2.6. I/O System

The basic model of the UNIX I/O system is a sequence of bytes that can be accessed either randomly or sequentially. There are no access methods and no control blocks in a typical UNIX user process.

Different programs expect various levels of structure, but the kernel does not impose structure on I/O. For instance, the convention for text files is lines of ASCII characters separated by a single newline character (the ASCII line-feed character), but the kernel knows nothing about this convention. For the purposes of most programs, the model is further simplified to being a stream of data bytes, or an I/O stream. It is this single common data form that makes the characteristic UNIX tool-based approach work [Kernighan & Pike, 1984]. An I/O stream from one program can be fed as input to almost any other program.

Descriptors and I/O

UNIX processes use descriptors to reference I/O streams. Descriptors are small unsigned integers obtained from the open and socket system calls. The open system call takes as arguments the name of a file and a permission mode to specify whether the file should be open for reading or for writing, or for both. This system call also can be used to create a new, empty file. A read or write system call can be applied to a descriptor to transfer data. The close system call can be used to deallocate any descriptor.

Descriptors represent underlying objects supported by the kernel and are created by system calls specific to the type of object. In FreeBSD, four kinds of objects can be represented by descriptors: files, pipes, fifos, and sockets.

  • A file is a linear array of bytes with at least one name. A file exists until all of its names are deleted explicitly and no process holds a descriptor for it. A process acquires a descriptor for a file by opening that file's name with the open system call. I/O devices are accessed as files.

  • A pipe is a linear array of bytes, as is a file, but it is used solely as an I/O stream, and it is unidirectional. It also has no name and thus cannot be opened with open. Instead, it is created by the pipe system call, which returns two descriptors, one of which accepts input that is sent to the other descriptor reliably, without duplication, and in order.

  • A fifo is often referred to as a named pipe. A fifo has properties identical to a pipe, except that it appears in the filesystem; thus, it can be opened using the open system call. Two processes that wish to communicate each open the fifo: One opens it for reading, the other for writing.

  • A socket is a transient object that is used for interprocess communication; it exists only as long as some process holds a descriptor referring to it. A socket is created by the socket system call, which returns a descriptor for it. There are different kinds of sockets that support various communication semantics, such as reliable delivery of data, preservation of message ordering, and preservation of message boundaries.

In systems before 4.2BSD, pipes were implemented using the filesystem; when sockets were introduced in 4.2BSD, pipes were reimplemented as sockets. For performance reasons, FreeBSD no longer uses sockets to implement pipes and fifos. Rather, it uses a separate implementation optimized for local communication.

The kernel keeps a descriptor table for each process, which is a table that the kernel uses to translate the external representation of a descriptor into an internal representation. (The descriptor is merely an index into this table.) The descriptor table of a process is inherited from that process's parent, and thus access to the objects to which the descriptors refer also is inherited. The main ways that a process can obtain a descriptor are

  1. by opening or creating an object and

  2. by inheriting from the parent process.

In addition, socket IPC allows passing descriptors in messages between unrelated processes on the same machine.

Every valid descriptor has an associated file offset in bytes from the beginning of the object. Read and write operations start at this offset, which is updated after each data transfer. For objects that permit random access, the file offset also may be set with the lseek system call. Ordinary files permit random access, and some devices do, too. Pipes, fifos, and sockets do not.

When a process terminates, the kernel reclaims all the descriptors that were in use by that process. If the process was holding the final reference to an object, the object's manager is notified so that it can do any necessary cleanup actions, such as final deletion of a file or deallocation of a socket.

Descriptor Management

Most processes expect three descriptors to be open already when they start running. These descriptors are 0, 1, and 2, more commonly known as standard input, standard output, and standard error, respectively. Usually, all three are associated with the user's terminal by the login process (see Section 14.5) and are inherited through fork and exec by processes run by the user. Thus, a program can read what the user types by reading standard input, and the program can send output to the user's screen by writing to standard output. The standard error descriptor also is open for writing and is used for error output, whereas standard output is used for ordinary output.

These (and other) descriptors can be mapped to objects other than the terminal; such mapping is called I/O redirection, and all the standard shells permit users to do it. The shell can direct the output of a program to a file by closing descriptor 1 (standard output) and opening the desired output file to produce a new descriptor 1. It can similarly redirect standard input to come from a file by closing descriptor 0 and opening the file.

Pipes allow the output of one program to be input to another program without rewriting or even relinking of either program. Instead of descriptor 1 (standard output) of the source program being set up to write to the terminal, it is set up to be the input descriptor of a pipe. Similarly, descriptor 0 (standard input) of the sink program is set up to reference the output of the pipe instead of the terminal keyboard. The resulting set of two processes and the connecting pipe is known as a pipeline. Pipelines can be arbitrarily long series of processes connected by pipes.

The open, pipe, and socket system calls produce new descriptors with the lowest unused number usable for a descriptor. For pipelines to work, some mechanism must be provided to map such descriptors into 0 and 1. The dup system call creates a copy of a descriptor that points to the same file-table entry. The new descriptor is also the lowest unused one, but if the desired descriptor is closed first, dup can be used to do the desired mapping. Care is required, however: If descriptor 1 is desired, and descriptor 0 happens also to have been closed, descriptor 0 will be the result. To avoid this problem, the system provides the dup2 system call; it is like dup, but it takes an additional argument specifying the number of the desired descriptor (if the desired descriptor was already open, dup2 closes it before reusing it).

Devices

Hardware devices have filenames and may be accessed by the user via the same system calls used for regular files. The kernel can distinguish a device special file or special file, and it can determine to what device it refers, but most processes do not need to make this determination. Terminals, printers, and tape drives are all accessed as though they were streams of bytes, like FreeBSD disk files. Thus, device dependencies and peculiarities are kept in the kernel as much as possible, and even in the kernel most of them are segregated in the device drivers.

Processes typically access devices through special files in the filesystem. I/O operations to these files are handled by kernel-resident software modules termed device drivers. Most network-communication hardware devices are accessible through only the interprocess-communication facilities and do not have special files in the filesystem name space, because the raw-socket interface provides a more natural interface than does a special file.

Device special files are created in the /dev filesystem by their device driver when the hardware is first found. The ioctl system call manipulates the underlying device parameters of special files. The operations that can be done differ for each device. This system call allows the special characteristics of devices to be accessed, rather than overloading the semantics of other system calls. For example, there is an ioctl on a sound card to set the audio-encoding format instead of there being a special or modified version of write.

Socket IPC

The 4.2BSD kernel introduced an IPC mechanism more flexible than pipes, based on sockets. A socket is an endpoint of communication referred to by a descriptor, just like a file or a pipe. Two processes can each create a socket and then connect those two endpoints to produce a reliable byte stream. Once connected, the descriptors for the sockets can be read or written by processes, just as the latter would do with a pipe. The transparency of sockets allows the kernel to redirect the output of one process to the input of another process residing on another machine. A major difference between pipes and sockets is that pipes require a common parent process to set up the communications channel. A connection between sockets can be set up by two unrelated processes, possibly residing on different machines.

Fifos appear as an object in the filesystem that unrelated processes can open and send data through in the same way as they would communicate through a pair of sockets. Thus, fifos do not require a common parent to set them up; they can be connected after a pair of processes are up and running. Unlike sockets, fifos can be used on only a local machine; they cannot be used to communicate between processes on different machines.

The socket mechanism requires extensions to the traditional UNIX I/O system calls to provide the associated naming and connection semantics. Rather than overloading the existing interface, the developers used the existing interfaces to the extent that the latter worked without being changed and designed new interfaces to handle the added semantics. The read and write system calls were used for byte-stream-type connections, but six new system calls were added to allow sending and receiving addressed messages such as network datagrams. The system calls for writing messages include send, sendto, and sendmsg. The system calls for reading messages include recv, recvfrom, and recvmsg. In retrospect, the first two in each class are special cases of the others; recvfrom and sendto probably should have been added as library interfaces to recvmsg and sendmsg, respectively.

Scatter/Gather I/O

In addition to the traditional read and write system calls, 4.2BSD introduced the ability to do scatter/gather I/O. Scatter input uses the readv system call to allow a single read to be placed in several different buffers. Conversely, the writev system call allows several different buffers to be written in a single atomic write. Instead of passing a single buffer and length parameter, as is done with read and write, the process passes in a pointer to an array of buffers and lengths, along with a count describing the size of the array.

This facility allows buffers in different parts of a process address space to be written atomically, without the need to copy them to a single contiguous buffer. Atomic writes are necessary in the case where the underlying abstraction is record based, such as datagrams that output a single message on each write request. It is also convenient to be able to read a single request into several different buffers (such as a record header into one place and the data into another). Although an application can simulate the ability to scatter data by reading the data into a large buffer and then copying the pieces to their intended destinations, the cost of memory-to-memory copying in such cases often would more than double the running time of the affected application.

Just as send and recv could have been implemented as library interfaces to sendto and recvfrom, it also would have been possible to simulate read using readv and write using writev. However, read and write are used so much more frequently that the added cost of simulating them would not have been worthwhile.

Multiple Filesystem Support

With the expansion of network computing, it became desirable to support both local and remote filesystems. To simplify the support of multiple filesystems, the developers added a new virtual node or vnode interface to the kernel. The set of operations exported from the vnode interface appear much like the filesystem operations previously supported by the local filesystem. However, they may be supported by a wide range of filesystem types.

  • Local disk-based filesystems

  • Files imported using a variety of remote filesystem protocols

  • Read-only CD-ROM filesystems

  • Filesystems providing special-purpose interfaces—for example, the /proc filesystem

By using loadable kernel modules (see Section 14.4), FreeBSD allows filesystems to be loaded dynamically when the filesystems are first referenced by the mount system call. The vnode interface is described in Section 6.5; its ancillary support routines are described in Section 6.6; several of the special-purpose filesystems are described in Section 6.7.


2.7. Devices

Historically, the device interface was static and simple. Devices were discovered as the system was booted and did not change thereafter. A typical disk driver could be written in a few hundred lines of code. As the system has evolved, the complexity of the I/O system has increased, with the addition of new functionality. Devices may appear and later disappear while the system is running. With increasing complexity and types of I/O buses, the routing of I/O requests has become complex. In a multiprocessor, for example, device interrupts must be routed to the most appropriate processor, which may not be the same one that previously handled the device. An overview of the PC architecture is given in Section 7.1.

Logical disks may no longer refer to a partition on a single physical disk but instead may combine several slices and/or partitions to create a virtual partition on which to build a filesystem that spans several disks. The aggregation of physical disk partitions into a virtual partition in these ways is referred to as volume management. Rather than building all this functionality into all the filesystems or disk drivers, it has been abstracted out into the GEOM (geometry) layer. The operation of the GEOM layer is described in Section 7.2. The management of the two major disk subsystems in FreeBSD is described in Sections 7.3 and 7.4.

Autoconfiguration is the procedure carried out by the system to recognize and enable the hardware devices present in a system. Historically, autoconfiguration was done just once when the system was booted. In current machines, particularly portable machines such as laptop computers, devices routinely come and go while the machine is operating. Thus, the kernel must be prepared to configure, initialize, and make available hardware when it arrives and to drop operations with hardware that has departed. FreeBSD uses a device-driver infrastructure called newbus to manage the devices on the system. The newbus architecture is described in Section 7.5.


2.8. Filesystems

A regular file is a linear array of bytes and can be read and written starting at any byte in the file. The kernel distinguishes no record boundaries in regular files, although many programs recognize line-feed characters as distinguishing the ends of lines, and other programs may impose other structure. No system-related information about a file is kept in the file itself, but the filesystem stores a small amount of ownership, protection, and usage information with each file.

A filename component is a string of up to 255 characters. These filenames are stored in a type of file called a directory. The information in a directory about a file is called a directory entry and includes, in addition to the filename, a pointer to the file itself. Directory entries may refer to other directories, as well as to plain files. A hierarchy of directories and files is thus formed, called a filesystem; a small one is shown in Figure 2.2. Directories may contain subdirectories, and there is no inherent limitation to the depth with which directory nesting may occur. To protect the consistency of the filesystem, the kernel does not permit processes to write directly into directories. A filesystem may include not only plain files and directories but also references to other objects, such as devices, sockets, and fifos.

Figure 2.2. A small filesystem tree.

The filesystem forms a tree, the beginning of which is the root directory, sometimes referred to by the name slash, spelled with a single solidus character (/). The root directory contains files; in our example in Figure 2.2, it contains kernel, a copy of the kernel-executable object file. It also contains directories; in this example, it contains the usr directory. Within the usr directory is the bin directory, which mostly contains executable object code of programs, such as the files ls and vi.

A process identifies a file by specifying that file's pathname, which is a string composed of zero or more filenames separated by slash (/) characters. The kernel associates two directories with each process for use in interpreting pathnames. A process's root directory is the topmost point in the filesystem that the process can access; it is ordinarily set to the root directory of the entire filesystem. A pathname beginning with a slash is called an absolute pathname and is interpreted by the kernel starting with the process's root directory.

A pathname that does not begin with a slash is called a relative pathname and is interpreted relative to the current working directory of the process. (This directory also is known by the shorter names current directory or working directory.) The current directory itself may be referred to directly by the name dot, spelled with a single period (.). The filename dot-dot (..) refers to a directory's parent directory. The root directory is its own parent.

A process may set its root directory with the chroot system call and its current directory with the chdir system call. Any process may do chdir at any time, but chroot is permitted only a process with superuser privileges. Chroot is normally used to set up restricted access to the system.

Using the filesystem shown in Figure 2.2, if a process has the root of the filesystem as its root directory and has /usr as its current directory, it can refer to the file vi either from the root with the absolute pathname /usr/bin/vi or from its current directory with the relative pathname bin/vi.

System utilities and databases are kept in certain well-known directories. Part of the well-defined hierarchy includes a directory that contains the home directory for each user—for example, /usr/staff/mckusick and /usr/staff/gnn in Figure 2.2. When users log in, the current working directory of their shell is set to the home directory. Within their home directories, users can create directories as easily as they can regular files. Thus, a user can build arbitrarily complex subhierarchies.

The user usually knows of only one filesystem, but the system may know that this one virtual filesystem is really composed of several physical filesystems, each on a different device. A physical filesystem may not span multiple logical devices. Since most physical disk devices are divided into several logical devices, there may be more than one filesystem per physical device, but there will be no more than one per logical device. One filesystem—the filesystem that anchors all absolute pathnames—is called the root filesystem and is always available. Others may be mounted—that is, they may be integrated into the directory hierarchy of the root filesystem. References to a directory that has a filesystem mounted on it are converted transparently by the kernel into references to the root directory of the mounted filesystem.

The link system call takes the name of an existing file and another name to create for that file. After a successful link, the file can be accessed by either filename. A filename can be removed with the unlink system call. When the final name for a file is removed (and the final process that has the file open closes it), the file is deleted.

Files are organized hierarchically in directories. A directory is a type of file, but, in contrast to regular files, a directory has a structure imposed on it by the system. A process can read a directory as it would an ordinary file, but only the kernel is permitted to modify a directory. Directories are created by the mkdir system call and are removed by the rmdir system call. Before 4.2BSD, the mkdir and rmdir system calls were implemented by a series of link and unlink system calls being done. There were three reasons for adding systems calls explicitly to create and delete directories.

  1. The operation could be made atomic. If the system crashed, the directory would not be left half-constructed, as could happen when a series of link operations were used.

  2. When a networked filesystem is being run, the creation and deletion of files and directories need to be specified atomically so that they can be serialized.

  3. When supporting non-UNIX filesystems, such as an NT filesystem, on another partition of the disk, the other filesystem may not support link operations. Although other filesystems might support the concept of directories, they probably would not create and delete the directories with links, as the UNIX filesystem does. Consequently, they could create and delete directories only if explicit directory create and delete requests were presented.

The chown system call sets the owner and group of a file, and chmod changes protection attributes. Stat applied to a filename can be used to read back such properties of a file. The fchown, fchmod, and fstat system calls are applied to a descriptor instead of to a filename to do the same set of operations. The rename system call can be used to give a file a new name in the filesystem, replacing one of the file's old names. Like the directory-creation and directory-deletion operations, the rename system call was added to 4.2BSD to provide atomicity to name changes in the local filesystem. Later, it proved useful explicitly to export renaming operations to foreign filesystems and over the network.

The truncate system call was added to 4.2BSD to allow files to be set to an arbitrary size. Thus, truncate is poorly named because it may be used to both shorten and lengthen a file. Files may have holes in them. Holes are void areas in the linear extent of the file where data have never been written. A process can create these holes by positioning the pointer past the current end-of-file and writing. Alternatively, a hole may be added to the end of a file by using the truncate system call to increase its length. When read, holes are treated by the system as zero-valued bytes.

Once the filesystem had the ability to shorten files, the kernel took advantage of that ability to shorten large, empty directories. The advantage of shortening empty directories is that it reduces the time spent in the kernel searching them when names are being created or deleted.

Newly created files are assigned the user identifier of the process that created them and the group identifier of the directory in which they were created. A three-level access-control mechanism is provided for the protection of files. The following three levels specify the accessibility of a file.

  1. To the user who owns the file

  2. To the group that owns the file

  3. To everybody else

Each level of access has separate indicators for read permission, write permission, and execute permission. If finer granularity access control is needed, FreeBSD 5.2 also provides ACLs (access control lists) to allow specification of read, write, and execute permission on a per-user or per-group level.

Files are created with zero length and may grow when they are written. While a file is open, the system maintains a pointer into the file showing the current location in the file associated with the descriptor. This pointer can be moved about in the file in a random-access fashion. Processes sharing a file descriptor through a fork or dup system call share the current location pointer. Descriptors created by separate open system calls have separate current location pointers.

Filestores

The user visible part of the filesystem is its hierarchical naming, locking, quotas, attribute management, and protection. But the bulk of the filesystem implementation involves the organization and management of the data on the storage media. Laying out the contents of files on the storage media is the responsibility of the filestore. FreeBSD uses the traditional Berkeley fast filesystem format. The disk is organized into groups of contiguous blocks called cylinder groups. Files that are likely to be accessed together, based on their locations in the filesystem hierarchy, are stored in the same cylinder group. Files that are not expected to be accessed together are moved into different cylinder groups.

A key responsibility of the filestore is to ensure that the filesystem is always kept in a state in which it can be recovered after a hardware or software failure. While recoverability can be maintained by using synchronous writes to the disk, the performance of a filesystem using this technique would be unacceptably slow. FreeBSD uses a technique called soft updates (see Section 8.6) to ensure recoverability while still delivering good performance and fast restart after a system crash.

Another useful feature of the FreeBSD filestore is the ability to quickly take a filesystem snapshot. Snapshots can be taken every few hours and mounted in a well-known location so that users can recover inadvertently deleted files that they created or wrote earlier in the day. Snapshots can also be used to allow the creation of consistent archives of filesystems that are in continuous active use. Snapshots are described in Section 8.7.


2.9. Network Filesystem

Initially, networking was used to transfer data from one machine to another. Later, it evolved to allowing users to log in remotely to another machine. The next logical step was to bring the data to the user, instead of having the user go to the data—and network filesystems were born. Users working locally do not experience the network delays on each keystroke, so they have a more responsive environment.

Bringing the filesystem to a local machine was among the first of the major client-server applications. The server is the remote machine that exports one or more of its filesystems. The client is the local machine that imports those filesystems. From the local client's point of view, a remotely mounted filesystem appears in the file-tree name space just like any other locally mounted filesystem. Local clients can change into directories on the remote filesystem and can read, write, and execute binaries within that remote filesystem identically to the way that they can do these operations on a local filesystem.

When the local client does an operation on a remote filesystem, the request is packaged and is sent to the server. The server does the requested operation and returns either the requested information or an error indicating why the request was denied. To get reasonable performance, the client must cache frequently accessed data. The complexity of remote filesystems lies in maintaining cache consistency between the server and its many clients.

Although many remote-filesystem protocols have been developed over the years, the most pervasive one in use among UNIX systems is the Network Filesystem (NFS), whose protocol and most widely used implementation were done by Sun Microsystems. The FreeBSD kernel supports the NFS protocol, although the implementation was done independently from the protocol specification [Macklem, 1994]. The NFS protocol is described in Chapter 9.


2.10. Terminals

Historically, users interacted with the system using terminals connected to the computer through hardwired lines. Although hardwired terminals are largely a footnote in history, the character processing done for keyboard I/O is still important. The most common type of user session in FreeBSD uses a pseudo-terminal. The pseudo-terminal is built from a device pair, termed the master and slave devices. The slave device provides to a process an interface identical to that historically provided by a hardware device. However, instead of being associated with a hardware device to read and write characters, the slave device has another process manipulating it through the master half of the pseudo-terminal. That is, anything written on the master device is provided to the slave device as input, and anything written on the slave device is presented to the master device as input. Pseudo-terminals are used by the X-windows terminal emulator, xterm, as well as the common remote-login programs such as ssh and telnet.

Terminals support the standard system I/O operations, as well as a collection of terminal-specific operations to control input-character editing and output delays. The character processing is handled by a line discipline. The default line discipline is selected when a terminal is being used for an interactive login. The line discipline is run in canonical mode; input is processed to provide standard line-oriented editing functions, and input is presented to a process on a line-by-line basis.

Screen editors and programs that communicate with other computers generally run in noncanonical mode (also commonly referred to as raw mode or character-at-a-time mode). In this mode, input is passed through to the reading process immediately and without interpretation. All special-character input processing is disabled, no erase or other line editing processing is done, and all characters are passed to the program that is reading from the terminal.

It is possible to configure the terminal in thousands of combinations between these two extremes. For example, a screen editor that wanted to receive user interrupts asynchronously might enable the special characters that generate signals and enable output flow control but otherwise run in noncanonical mode. All other characters would be passed through to the process uninterpreted.

On output, the terminal handler provides simple formatting services, including the following.

  • Converting the line-feed character to the two-character carriage-return-line-feed sequence

  • Expanding tabs

  • Displaying echoed nongraphic ASCII characters as a two-character sequence of the form "^C" (i.e., the ASCII caret character followed by the ASCII character that is the character's value offset from the ASCII "@" character).

Each of these formatting services can be disabled individually by a process through control requests.


2.11. Interprocess Communication

Interprocess communication in FreeBSD is organized in communication domains. The most important domains currently supported include the local domain, for communication between processes executing on the same machine; the IPv4 domain, for communication between processes using the TCP/IP protocol suite (version 4); and the IPv6 domain, which is the newest version of the Internet protocols.

Within a domain, communication takes place between communication end-points known as sockets. As mentioned in Section 2.6, the socket system call creates a socket and returns a descriptor; other IPC system calls are described in Chapter 11. Each socket has a type that defines its communications semantics; these semantics include properties such as reliability, ordering, and prevention of duplication of messages.

Each socket has associated with it a communication protocol. This protocol provides the semantics required by the socket according to the latter's type. Applications may request a specific protocol when creating a socket or may allow the system to select a protocol that is appropriate for the type of socket being created.

Sockets may have addresses bound to them. The form and meaning of socket addresses are dependent on the communication domain in which the socket is created. Binding a name to a socket in the local domain causes a file to be created in the filesystem.

Normal data transmitted and received through sockets are untyped. Data-representation issues are the responsibility of libraries built on top of the interprocess-communication facilities.

Networking implementations on UNIX before 4.2BSD usually worked by overloading the character-device interfaces. One goal of the socket interface was for naive programs to be able to work without change on stream-style connections. Such programs can work only if the read and write systems calls are unchanged. Consequently, the original interfaces were left intact and were made to work on stream-type sockets. A new interface was added for more complicated sockets, such as those used to send datagrams, with which a destination address must be presented with each send call.

Another benefit is that the new interface is highly portable. Implementations of the sockets API exist for pretty much every modern operating system, including several that differ greatly from UNIX.

FreeBSD also supports several local IPC mechanisms not related to networking, including semaphores, message queues, and shared memory. These mechanisms are covered in Section 11.8.


2.12. Network Communication

Some of the communication domains supported by the socket IPC mechanism provide access to network protocols. These protocols are implemented as a separate software layer logically below the socket software in the kernel. The kernel provides many ancillary services, such as buffer management, message routing, standardized interfaces to the protocols, and interfaces to the network interface drivers for the use of the various network protocols.

At present there are two different versions of the Internet protocols in use, one established and the other under development. BSD has supported multiple protocols since 4.2BSD, providing interoperability and resource sharing among the diverse set of machines that exist in the Internet. Multiple-protocol support also provides for future changes. Today's protocols designed for 10- to 100-Mbit-per-second Ethernets are likely to be inadequate for tomorrow's 10- to 100-Gbit-per-second networks. Consequently, the network-communication layer is designed to support multiple protocols. New protocols are added to the kernel without the support for older protocols being affected. Older applications can continue to operate using the old protocol over the same physical network as is used by newer applications running with a newer network protocol.


2.13. Network Implementation

The first protocol suite implemented in 4.2BSD was DARPA's Transmission Control Protocol/Internet Protocol (IPv4). The CSRG chose TCP/IP as the first network to incorporate into the socket IPC framework, because a 4.1BSD-based implementation was publicly available from a DARPA-sponsored project at Bolt, Beranek, and Newman (BBN). That was an influential choice: The 4.2BSD implementation is the main reason for the extremely widespread use of this protocol suite. Later performance and functional improvements to the TCP/IP implementation have also been widely adopted. The TCP/IP implementation is described in detail in Chapter 13.

The release of FreeBSD 4.0 added support for the IPv6 protocol suite, built by the KAME group in Japan. This suite is the next generation of Internet protocols that will eventually supplant the IPv4 Internet that was made popular partly by BSD.


2.14. System Operation

Bootstrapping mechanisms are used to start the system running. First, the FreeBSD kernel must be loaded into the main memory of the processor. Once loaded, it must go through an initialization phase to set the hardware into a known state. Next, the kernel must do device-dependent and device-independent initialization. The system begins running in single-user mode while a startup script does disk checks and starts the accounting and quota checking. Finally, the startup script starts the general system services and brings up the system to full multiuser operation.

In multiuser operation the system may act as a general timesharing system, supporting direct or network-based logins by users who then start processes running on their behalf. FreeBSD often acts as a server, providing file services and serving Web requests to network clients. All these network-based services can be started automatically at boot time. When used as a server, there is rarely more than one human user logged into the system (the administrator).


Exercises

2.1 How does a user process request a service from the kernel?

2.2 How are data transferred between a process and the kernel? What alternatives are available?

2.3 How does a process access an I/O stream? List three types of I/O streams.

2.4 What are the four steps in the lifecycle of a process?

2.5 Why are process groups provided in FreeBSD?

2.6 Describe four machine-dependent functions of the kernel?

2.7 Describe the difference between an absolute and a relative pathname.

2.8 Give three reasons why the mkdir system call was added to 4.2BSD.

2.9 Define scatter-gather I/O. Why is it useful?

2.10 List five functions provided by a terminal driver.

2.11 What is the difference between a pipe and a socket?

2.12 Describe how to create a group of processes in a pipeline.

*2.13 List the three system calls that were required to create a new directory foo in the current directory before the addition of the mkdir system call.

*2.14 Explain the difference between interprocess communication and networking.


References

Accetta et al., 1986. M. Accetta, R. Baron, W. Bolosky, D. Golub, R. Rashid, A. Tevanian, & M. Young, "Mach: A New Kernel Foundation for UNIX Development," USENIX Association Conference Proceedings, pp. 93–113, June 1986.

Cheriton, 1988. D. R. Cheriton, "The V Distributed System," Comm ACM, vol. 31, no. 3, pp. 314–333, March 1988.

Ewens et al., 1985. P. Ewens, D. R. Blythe, M. Funkenhauser, & R. C. Holt, "Tunis: A Distributed Multiprocessor Operating System," USENIX Association Conference Proceedings, pp. 247–254, June 1985.

Gingell et al., 1987. R. Gingell, J. Moran, & W. Shannon, "Virtual Memory Architecture in SunOS," USENIX Association Conference Proceedings, pp. 81–94, June 1987.

Kernighan & Pike, 1984. B. W. Kernighan & R. Pike, The UNIX Programming Environment, Prentice-Hall, Englewood Cliffs, NJ, 1984.

Macklem, 1994. R. Macklem, "The 4.4BSD NFS Implementation," in 4.4BSD System Manager's Manual, pp. 6:1–14, O'Reilly & Associates, Inc., Sebastopol, CA, 1994.

McKusick & Karels, 1988. M. K. McKusick & M. Karels, "Design of a General Purpose Memory Allocator for the 4.3BSD UNIX Kernel," USENIX Association Conference Proceedings, pp. 295–304, June 1988.

McKusick et al., 1994. M. K. McKusick, M. Karels, S. J. Leffler, W. N. Joy, & R. S. Fabry, "Berkeley Software Architecture Manual, 4.4BSD Edition," in 4.4BSD Programmer's Supplementary Documents, pp. 5:1–42, O'Reilly & Associates, Inc., Sebastopol, CA, 1994.

Ritchie, 1988. D. M. Ritchie, "Early Kernel Design," private communication, March 1988.

Rozier et al., 1988. M. Rozier, V. Abrossimov, F. Armand, I. Boule, M. Gien, M. Guillemont, F. Herrmann, C. Kaiser, S. Langlois, P. Leonard, & W. Neuhauser, "Chorus Distributed Operating Systems," USENIX Computing Systems, vol. 1, no. 4, pp. 305–370, Fall 1988.

Tevanian, 1987. A. Tevanian, "Architecture-Independent Virtual Memory Management for Parallel and Distributed Environments: The Mach Approach," Technical Report CMU-CS-88-106, Department of Computer Science, Carnegie-Mellon University, Pittsburgh, PA, December 1987.


Chapter 3. Kernel Services


3.1. Kernel Organization

The FreeBSD kernel can be viewed as a service provider to user processes. Processes usually access these services through system calls. Some services, such as process scheduling and memory management, are implemented as processes that execute in kernel mode or as routines that execute periodically within the kernel. In this chapter, we describe how kernel services are provided to user processes, and some of the ancillary processing done by the kernel. Then we describe the basic kernel services provided by FreeBSD and provide details of their implementation.

System Processes

All FreeBSD user-level processes originate from a single process that is crafted by the kernel at startup. Table 3.1 (on page 50) lists the processes that are created immediately and exist always. They are kernel processes, and they function wholly within the kernel. (Kernel processes execute code that is compiled into the kernel's load image and operate with the kernel's privileged execution mode.) The kernel also starts a kernel process for each device to handle interrupts for that device.

Table 3.1. Permanent kernel processes.

Name

Description

idle

runs when there is nothing else to do

swapper

schedules the loading of processes in main memory from secondary storage when system resources become available

vmdaemon

schedules the transfer of whole processes from main memory to secondary storage when system resources are low

pagedaemon

writes parts of the address space of a process to secondary storage to support the paging facilities of the virtual-memory system

pagezero

maintains a supply of zero'ed pages

bufdaemon

maintains a supply of clean buffers by writing out dirty buffers when the supply of clean buffers gets too low

syncer

ensures that dirty file data is written after 30 seconds

ktrace

writes system call tracing records to their output file

vnlru

maintains a supply of free vnodes by cleaning up the least recently used ones

random

collects entropy data to supply seeding for kernel random numbers and the /dev/random device

g_event

handle configuration tasks including discovery of new devices and cleanup of devices when they disappear

g_up

handle data coming from device drivers and being delivered to processes

g_down

handle data coming from processes and being delivered to device drivers

After creating the kernel processes, the kernel creates the first process to execute a program in user mode; it serves as the parent process for all subsequent processes. The first user-mode process is the init process—historically, process 1. This process does administrative tasks, such as spawning getty processes for each terminal on a machine, collecting exit status from orphaned processes, and handling the orderly shutdown of a system from multiuser to single-user operation. The init process is a user-mode process, running outside the kernel (see Section 14.6).

System Entry

Entrances into the kernel can be categorized according to the event or action that initiates it.

  • Hardware interrupt

  • Hardware trap

  • Software-initiated trap

Hardware interrupts arise from external events, such as an I/O device needing attention or a clock reporting the passage of time. (For example, the kernel depends on the presence of a real-time clock or interval timer to maintain the current time of day, to drive process scheduling, and to initiate the execution of system timeout functions.) Hardware interrupts occur asynchronously and may not relate to the context of the currently executing process.

Hardware traps may be either synchronous or asynchronous but are related to the current executing process. Examples of hardware traps are those generated as a result of an illegal arithmetic operation, such as divide by zero.

Software-initiated traps are used by the system to force the scheduling of an event, such as process rescheduling or network processing, as soon as possible. Software-initiated traps are implemented by setting a flag that is checked whenever a process is preparing to exit from the kernel. If the flag is set, the software-interrupt code is run instead of exiting from the kernel.

System calls are a special case of a software-initiated trap—the machine instruction used to initiate a system call typically causes a hardware trap that is handled specially by the kernel.

Run-Time Organization

The kernel can be logically divided into a top half and a bottom half, as shown in Figure 3.1. The top half of the kernel provides services to processes in response to system calls or traps. This software can be thought of as a library of routines shared by all processes. The top half of the kernel executes in a privileged execution mode, in which it has access both to kernel data structures and to the context of user-level processes. The context of each process is contained in two areas of memory reserved for process-specific information. The first of these areas is the process structure, which has historically contained the information that is necessary even if the process has been swapped out. In FreeBSD, this information includes the identifiers associated with the process, the process's rights and privileges, its descriptors, its memory map, pending external events and associated actions, maximum and current resource utilization, and many other things. The second is the user structure, which has historically contained the information that is not necessary when the process is swapped out. In FreeBSD, the user-structure information of each process includes the hardware thread control block (TCB), process accounting and statistics, and minor additional information for debugging and creating a core dump. Deciding what was to be stored in the process structure and the user structure was far more important in previous systems than it was in FreeBSD. As memory became a less limited resource, most of the user structure was merged into the process structure for convenience; see Section 4.2.

Figure 3.1. Run-time structure of the kernel.

The bottom half of the kernel comprises routines that are invoked to handle hardware interrupts. Activities in the bottom half of the kernel are synchronous with respect to the interrupt source but are asynchronous, with respect to the top half, and the software cannot depend on having a specific (or any) process running when an interrupt occurs. Thus, the state information for the process that initiated the activity is not available. The top and bottom halves of the kernel communicate through data structures, generally organized around work queues.

The FreeBSD kernel is never preempted to run another user process while executing in the top half of the kernel—for example, while executing a system call—although it will explicitly give up the processor if it must wait for an event or for a shared resource. Its execution may be interrupted, however, by interrupts for the bottom half of the kernel. When an interrupt is received, the kernel process that handles that device is scheduled to run. Normally these device-interrupt processes have a higher priority than user processes or processes running in the top half of the kernel. Thus, when an interrupt causes a device-interrupt process to be made runnable, it will usually preempt the currently running process. When a process running in the top half of the kernel wants to add an entry to the work list for a device, it needs to ensure that it will not be preempted by that device part way through linking the new element onto the work list. In FreeBSD 5.2, the work list is protected by a mutex. Any process (top or bottom half) seeking to modify the work list must first obtain the mutex. Once held, any other process seeking to obtain the mutex will wait until the process holding it has finished modifying the list and released the mutex.

Processes cooperate in the sharing of system resources, such as the disks and memory. The top and bottom halves of the kernel also work together in implementing certain system operations, such as I/O. Typically, the top half will start an I/O operation, and then relinquish the processor; then the requesting process will sleep, awaiting notification from the bottom half that the I/O request has completed.

Entry to the Kernel

When a process enters the kernel through a trap or an interrupt, the kernel must save the current machine state before it begins to service the event. For the PC, the machine state that must be saved includes the program counter, the user stack pointer, the general-purpose registers, and the processor status longword. The PC trap instruction saves the program counter and the processor status longword as part of the exception stack frame; the user stack pointer and registers must be saved by the software trap handler. If the machine state were not fully saved, the kernel could change values in the currently executing program in improper ways. Since interrupts may occur between any two user-level instructions (and on some architectures between parts of a single instruction), and because they may be completely unrelated to the currently executing process, an incompletely saved state could cause correct programs to fail in mysterious and not easily reproducible ways.

The exact sequence of events required to save the process state is completely machine dependent, although the PC provides a good example of the general procedure. A trap or system call will trigger the following events:

  • The hardware switches into kernel (supervisor) mode, so that memory-access checks are made with kernel privileges, references to the stack use the per-process kernel stack, and privileged instructions can be executed.

  • The hardware pushes onto the per-process kernel stack the program counter, processor status longword, and information describing the type of trap. (On architectures other than the PC, this information can include the system-call number and general-purpose registers as well.)

  • An assembly-language routine saves all state information not saved by the hardware. On the PC, this information includes the general-purpose registers and the user stack pointer, also saved onto the per-process kernel stack.

After this preliminary state saving, the kernel calls a C routine that can freely use the general-purpose registers as any other C routine would, without concern about changing the unsuspecting process's state.

There are three major kinds of handlers, corresponding to particular kernel entries:

  1. Syscall() for a system call

  2. Trap() for hardware traps and for software-initiated traps other than system calls

  3. The appropriate device-driver interrupt handler for a hardware interrupt

Each type of handler takes its own specific set of parameters. For a system call, they are the system-call number and an exception frame. For a trap, they are the type of trap, the relevant floating-point and virtual-address information related to the trap, and an exception frame. (The exception-frame arguments for the trap and system call are not the same. The PC hardware saves different information based on different types of traps.) For a hardware interrupt, the only parameter is a unit (or board) number.

Return from the Kernel

When the handling of the system entry is completed, the user-process state is restored, and control returns to the user process. Returning to the user process reverses the process of entering the kernel.

  • An assembly-language routine restores the general-purpose registers and user-stack pointer previously pushed onto the stack.

  • The hardware restores the program counter and program status longword, and switches to user mode, so that future references to the stack pointer use the user's stack pointer, privileged instructions cannot be executed, and memory-access checks are done with user-level privileges.

Execution then resumes at the next instruction in the user's process.


3.2. System Calls

The most frequent trap into the kernel (after clock processing) is a request to do a system call. System performance requires that the kernel minimize the overhead in fielding and dispatching a system call. The system-call handler must do the following work:

  • Verify that the parameters to the system call are located at a valid user address, and copy them from the user's address space into the kernel

  • Call a kernel routine that implements the system call

Result Handling

Eventually, the system call returns to the calling process, either successfully or unsuccessfully. On the PC architecture, success or failure is returned as the carry bit in the user process's program status longword: If it is zero, the return was successful; otherwise, it was unsuccessful. On many machines, return values of C functions are passed back through a general-purpose register (for the PC, data register EAX). The routines in the kernel that implement system calls return the values that are normally associated with the global variable errno. After a system call, the kernel system-call handler leaves this value in the register. If the system call failed, a C library routine moves that value into errno, and sets the return register to -1. The calling process is expected to notice the value of the return register, and then to examine errno. The mechanism involving the carry bit and the global variable errno exists for historical reasons derived from the PDP-11.

There are two kinds of unsuccessful returns from a system call: those where kernel routines discover an error and those where a system call is interrupted. The most common case is a system call that is interrupted when it has relinquished the processor to wait for an event that may not occur for a long time (such as terminal input), and a signal arrives in the interim. When signal handlers are initialized by a process, they specify whether system calls that they interrupt should be restarted or whether the system call should return with an interrupted system call (EINTR) error.

When a system call is interrupted, the signal is delivered to the process. If the process has requested that the signal abort the system call, the handler then returns an error, as described previously. If the system call is to be restarted, however, the handler resets the process's program counter to the machine instruction that caused the system-call trap into the kernel. (This calculation is necessary because the program-counter value that was saved when the system-call trap was done is for the instruction after the trap-causing instruction.) The handler replaces the saved program-counter value with this address. When the process returns from the signal handler, it resumes at the program-counter value that the handler provided and reexecutes the same system call.

Restarting a system call by resetting the program counter has certain implications. First, the kernel must not modify any of the input parameters in the process address space (it can modify the kernel copy of the parameters that it makes). Second, it must ensure that the system call has not performed any actions that cannot be repeated. For example, in the current system, if any characters have been read from the terminal, the read must return with a short count. Otherwise, if the call were to be restarted, the already-read bytes would be lost.

Returning from a System Call

While the system call is running or sleeping with signals blocked, a signal may be posted to the process, or another process may attain a higher scheduling priority. After the system call completes, the system-call exit code checks to see whether either event has occurred.

The system-call exit code first checks for a posted signal. Such signals include signals that interrupted the system call, as well as signals that arrived while a system call was in progress but were held pending until the system call completed. Signals that are ignored, by default or by explicit programmatic request, are never posted to the process. Signals with a default action have that action taken before the process runs again (i.e., the process may be stopped or terminated as appropriate). If a signal is to be caught (and is not currently blocked), the system-call exit code arranges to have the appropriate signal handler called rather than have the process return directly from the system call. After the signal handler returns, the process will resume execution at system-call return (or system-call execution, if the system call is being restarted).

After checking for posted signals, the system-call exit code checks to see whether any process has a priority higher than that of the currently running one. If such a process exists, the system-call exit code calls the context-switch routine to cause the higher-priority process to run. At a later time, the current process will again have the highest priority and will resume execution by returning from the system call to the user process.

If a process has requested that the system do profiling, the system-call exit code also calculates the amount of time that has been spent in the system call— that is, the system time accounted to the process between the latter's entry into and exit from the kernel. This time is charged to the routine in the user's process that made the system call.


3.3. Traps and Interrupts

Traps

Traps, like system calls, occur synchronously for a process. Traps normally occur because of unintentional errors, such as division by zero or indirection through an invalid pointer. The process becomes aware of the problem either by catching a signal or by being terminated. Traps can also occur because of a page fault, in which case the system makes the page available and restarts the process without the process being aware that the fault occurred.

The trap handler is invoked like the system-call handler. First, the process state is saved. Next, the trap handler determines the trap type and then arranges to post a signal or to cause a pagein as appropriate. Finally, it checks for pending signals and higher-priority processes, and exits like the system-call handler except that it has no return value.

I/O Device Interrupts

Interrupts from I/O and other devices are handled by interrupt routines that are loaded as part of the kernel's address space. These routines handle the console terminal interface, one or more clocks, and several software-initiated interrupts used by the system for low-priority clock processing and for networking facilities.

Unlike traps and system calls, device interrupts occur asynchronously. The process that requested the service is unlikely to be the currently running process and may no longer exist! The process that started the operation will be notified that the operation has finished when that process runs again. As occurs with traps and system calls, the entire machine state must be saved, since any changes could cause errors in the currently running process.

Device-interrupt handlers run only on demand. Unlike earlier versions of FreeBSD, the FreeBSD 5.2 kernel creates a context for each device driver. Thus, just as one process cannot access the context of the previously running process, interrupt handlers cannot access any of the context of the previously running interrupt handler. The stack normally used by the kernel is part of a process context. Since each device has its own context, it also has its own stack on which to run.

Interrupts in earlier FreeBSD systems had no context, so they had to run to completion without sleeping. In FreeBSD 5.2 interrupts can block to wait for resources. However, while blocked they cannot be invoked with another event, so to reduce the chance for lost interrupts, most handlers still run to completion without sleeping.

An interrupt handler is never invoked from the top half of the kernel. Thus, it must get all the information it needs from the data structures that it shares with the top half of the kernel—generally, its global work queue. Similarly, all information provided to the top half of the kernel by the interrupt handler must be communicated the same way.

Software Interrupts

Many events in the kernel are driven by hardware interrupts. For high-speed devices such as network controllers, these interrupts are scheduled at a high priority. A network controller must quickly acknowledge receipt of a packet and reen-able the controller to accept more packets to avoid losing closely spaced packets. However, the further processing of passing the packet to the receiving process, although time-consuming, does not need to be done quickly. Thus, the further processing can be scheduled at a lower priority, so critical operations will not be blocked from executing longer than necessary.

The mechanism for doing lower-priority processing is called a software interrupt. Typically, a high-priority interrupt creates a queue of work to be done at a lower-priority level. As with hardware devices in FreeBSD 5.2, each software interrupt has a process context associated with it. The software-interrupt processes are generally given a lower scheduling priority than the device-driver processes but a higher priority than those given to user processes. Whenever a hardware interrupt arrives, the process associated with device driver will attain the highest priority and be scheduled to run. When there are no device-driver processes that are runnable, the highest priority software-interrupt process will be scheduled to run. If there are no software-interrupt processes that are runnable, then the highest priority user process will run. If either a software-interrupt process or a user process is running when an interrupt arrives and makes its device-driver process runnable, the scheduler will preempt the software-interrupt or user process to run the device-driver process.

The delivery of network packets to destination processes is handled by a packet-processing function that runs at a lower priority than the network-controller device driver. As packets come in, they are put onto a work queue, and the controller is immediately reenabled. Between packet arrivals, the packet-processing process works to deliver the packets. Thus, the controller can accept new packets without having to wait for the previous packet to be delivered. In addition to network processing, software interrupts are used to handle time-related events and process rescheduling.


3.4. Clock Interrupts

The system is driven by a clock that interrupts at regular intervals. Each interrupt is referred to as a tick. On the PC, the clock ticks 100 times per second. At each tick, the system updates the current time of day as well as user-process and system timers.

Interrupts for clock ticks are posted at a high hardware-interrupt priority. After switching to the clock-device process, the hardclock() routine is called. It is important that the hardclock() routine finish its job quickly:

  • If hardclock() runs for more than one tick, it will miss the next clock interrupt. Since hardclock() maintains the time of day for the system, a missed interrupt will cause the system to lose time.

  • Because of hardclock()'s high interrupt priority, nearly all other activity in the system is blocked while hardclock() is running. This blocking can cause network controllers to miss packets or a disk controller to miss the transfer of a sector coming under a disk drive's head.

So the time spent in hardclock() is minimized, less critical time-related processing is handled by a lower-priority software-interrupt handler called softclock(). In addition, if multiple clocks are available, some time-related processing can be handled by other routines supported by alternate clocks. On the PC there are two additional clocks that run at a different frequency than the system clock: the statclock(), which runs at 128 ticks per second to collect system statistics, and the profclock(), which runs at 1024 ticks per second to collect profiling information.

The work done by hardclock() is as follows:

  • If the currently running process has a virtual or profiling interval timer (see Section 3.6), decrement the timer and deliver a signal if the timer has expired.

  • Increment the current time of day.

  • If the system does not have a separate clock for process profiling, the hardclock() routine does the operations normally done by profclock(), as described in the next section.

  • If the system does not have a separate clock for statistics gathering, the hardclock() routine does the operations normally done by statclock(), as described in the next section.

  • If softclock() needs to be run, make the softclock process runnable.

Statistics and Process Scheduling

On historic FreeBSD systems, the hardclock() routine collected resource-utilization statistics about what was happening when the clock interrupted. These statistics were used to do accounting, to monitor what the system was doing, and to determine future scheduling priorities. In addition, hardclock() forced context switches so that all processes would get a share of the CPU.

This approach has weaknesses because the clock supporting hardclock() interrupts on a regular basis. Processes can become synchronized with the system clock, resulting in inaccurate measurements of resource utilization (especially CPU) and inaccurate profiling [McCanne & Torek, 1993]. It is also possible to write programs that deliberately synchronize with the system clock to outwit the scheduler.

On architectures with multiple high-precision, programmable clocks—such as the PC—a statistics clock is run at a different frequency than the time-of-day clock. The FreeBSD statclock() runs at 128 ticks per second and is responsible for accumulating resource usage to processes. At each tick, it charges the currently running process with a tick; if the process has accumulated four ticks, recalculate its priority. If the new priority is less than the current priority, arrange for the process to be rescheduled. Thus, processes that become synchronized with the system clock still have CPU time accounted to them.

The statclock() also collects statistics on what the system was doing at the time of the tick (sitting idle, executing in user mode, or executing in system mode). Finally, it collects basic information on system I/O, such as which disk drives are currently active.

To allow the collection of more accurate profiling information, FreeBSD supports a profiling clock. When one or more processes are requesting profiling information, the profiling clock is set to run at a tick rate that is relatively prime to the main system clock (1024 ticks per second on the PC). At each tick, it checks to see if one of the processes that it has been asked to profile is running. If so, it obtains the current location of the program counter and increments a counter associated with that location in the profiling buffer associated with the process.

Timeouts

The remaining time-related processing involves processing timeout requests and periodically reprioritizing processes that are ready to run. These functions are handled by the softclock() routine.

When hardclock() completes, if there were any softclock() functions to be done, hardclock() schedules the softclock process to run.

The primary task of the softclock() routine is to arrange for the execution of periodic events, such as the following:

  • Process real-time timer (see Section 3.6)

  • Retransmission of dropped network packets

  • Watchdog timers on peripherals that require monitoring

  • System process-rescheduling events

An important event is the scheduling that periodically raises or lowers the CPU priority for each process in the system based on that process's recent CPU usage (see Section 4.4). The rescheduling calculation is done once per second. The scheduler is started at boot time, and each time that it runs, it requests that it be invoked again 1 second in the future.

On a heavily loaded system with many processes, the scheduler may take a long time to complete its job. Posting its next invocation 1 second after each completion may cause scheduling to occur less frequently than once per second. However, as the scheduler is not responsible for any time-critical functions, such as maintaining the time of day, scheduling less frequently than once a second is normally not a problem.

The data structure that describes waiting events is called the callout queue. Figure 3.2 shows an example of the callout queue. When a process schedules an event, it specifies a function to be called, a pointer to be passed as an argument to the function, and the number of clock ticks until the event should occur.

Figure 3.2. Timer events in the callout queue.

[View full size image]

The kernel maintains an array of queue headers, each representing a particular time. There is a pointer that references the queue header for the current time, marked "now" in Figure 3.2. The queue header that follows the currently referenced one represents events that are one tick in the future. The queue header after that is two ticks in the future. The list wraps around, so if the last queue header in the list represents time t, then the first queue header in the list represents time t + 1. The queue header immediately preceding the currently referenced one represents the time furthest in the future. In Figure 3.2 there are 200 queue headers, so the queue header immediately preceding the one marked "now" represents events that are 199 ticks in the future.

Each time the hardclock() routine runs, it increments the callout queue-head pointer. If the queue is not empty, it schedules the softclock() process to run. The softclock() process scans the events in the current queue. It compares the current time to the time stored in the event structure. If the times match, the event is removed from the list and its requested function is called being passed the specified argument.

When an event n ticks in the future is to be posted, its queue header is calculated by taking the index of the queue labeled "now" in Figure 3.2, adding n to it and then taking the resulting value modulo the number of queue headers. If an event is to occur further in the future than the number of queue headers, then it will end up on a list with other events that are to happen sooner. Thus, the actual time of the event is stored in its entry so that when the queue is scanned by softclock(), it can determine which events are current and which are to occur in the future. In Figure 3.2, the second entry in the "now" queue will be skipped on the current scan of the queue, but it will be handled 200 ticks in the future when softclock() next processes this queue.

An argument is provided to the callout-queue function when it is called so that one function can be used by multiple processes. For example, there is a single real-time timer function that sends a signal to a process when a timer expires. Every process that has a real-time timer running posts a timeout request for this function; the argument that is passed to the function is a pointer to the process structure for the process. This argument enables the timeout function to deliver the signal to the correct process.

Timeout processing is more efficient when the timeouts are specified in ticks. Time updates require only an integer decrement, and checks for timer expiration require only an integer comparison. If the timers contained time values, decrementing and comparisons would be more complex. The approach used in FreeBSD is based on the work of [Varghese & Lauck, 1987]. Another possible approach is to maintain a heap with the next-occurring event at the top [Barkley & Lee, 1988].


3.5. Memory-Management Services

The memory organization and layout associated with a FreeBSD process is shown in Figure 3.3 (on page 62). Each process begins execution with three memory segments: text, data, and stack. The data segment is divided into initialized data and uninitialized data (also known as bss). The text is read-only and is normally shared by all processes executing the file, whereas the data and stack areas can be written by, and are private to, each process. The text and initialized data for the process are read from the executable file.

Figure 3.3. Layout of a FreeBSD process in memory and on disk.

An executable file is distinguished by being a plain file (rather than a directory, special file, or symbolic link) and by having one or more of its execute bits set. Each executable file has an exec header containing a magic number that specifies the type of the executable file. FreeBSD supports multiple executable formats including the following:

  1. Files that must be read by an interpreter

  2. Files that are directly executable including AOUT, ELF, and gzipped ELF

An executable is initially parsed by the image activation (imgact) framework. The header of a file to be executed is passed through a list of registered image activators to find a matching format. When a matching format is found, the corresponding image activator prepares the file for execution.

Files falling into the first classification have as their magic number (located in the first 2 bytes of the file) the two-character sequence #! followed by the pathname of the interpreter to be used. This pathname is currently limited by a compile-time constant to 128 characters. For example, #!/bin/sh refers to the Bourne shell. The image activator that will be selected is the one that handles the invocation of interpreters. It will load and run the named interpreter, passing the name of the file that is to be interpreted as an argument. To prevent loops, FreeBSD allows only one level of interpretation, and a file's interpreter may not itself be interpreted.

For performance reasons, most files fall into the second classification and are directly executable. Information in the header of a directly executable file includes the architecture and operating system for which an executable was built and whether it is statically linked or uses shared libraries. The selected image activator can use information such as knowledge of the operating system for which an executable was compiled to configure the kernel to use the proper system call interpretation when running the program. For example, a executable built to run on Linux can be seamlessly run on FreeBSD by using the system-call dispatch-vector that provides emulation of the Linux system calls.

The header also specifies the sizes of text, initialized data, uninitialized data, and additional information for debugging. The debugging information is not used by the kernel or by the executing program. Following the header is an image of the text, followed by an image of the initialized data. Uninitialized data are not contained in the executable file because they can be created on demand using zero-filled memory.

To begin execution, the kernel arranges to have the text portion of the file mapped into the low part of the process address space starting at the beginning of the second page of the virtual address space. The first page of the virtual address space is marked as invalid so that attempts to read or write through a null pointer will fault. The initialized data portion of the file is mapped into the address space following the text. An area equal to the uninitialized data region is created with zero-filled memory after the initialized data region. The stack is also created from zero-filled memory. Although the stack should not need to be zero filled, early UNIX systems made it so. In an attempt to save some startup time in 4.2BSD, the developers modified the kernel to not zero-fill the stack, leaving the random previous contents of the page instead. But concerns about surreptitious misuse of data from previously running programs and unrepeatable errors in previously working programs lead to restoration of the zero filling of the stack by the time that 4.3BSD was released.

Copying into memory the entire text and initialized data portion of a large program causes a long startup latency. FreeBSD avoids this startup time by demand paging the program into memory rather than preloading the program. In demand paging, the program is loaded in small pieces (pages) as it is needed rather than all at once before it begins execution. The system does demand paging by dividing up the address space into equal-sized areas called pages. For each page, the kernel records the offset into the executable file of the corresponding data. The first access to an address on each page causes a page-fault trap in the kernel. The page-fault handler reads the correct page of the executable file into the process memory. Thus, the kernel loads only those parts of the executable file that are needed. Chapter 5 explains paging details.

It might seem more efficient to load the whole process at once rather than in many little pieces. However, most processes use less than half of their address space during their entire execution lifetime. The reason for the low utilization is that typical user commands have many options, only a few of which are used on any invocation. The code and data structures that support the unused options are not needed. Thus, the cost of loading the subset of pages that are used is lower than the cost of initially loading the whole process. In addition to the time saved by avoiding the loading of the entire process, demand paging also reduces the amount of physical memory that is needed to run the process.

The uninitialized data area can be extended with zero-filled pages using the system call sbrk, although most user processes use the library routine malloc(), a more programmer-friendly interface to sbrk. This allocated memory, which grows from the top of the original data segment, is called the heap. On the PC, the stack grows down from the top of memory, whereas the heap grows up from the bottom of memory.

Above the user stack are areas of memory that are created by the system when the process is started. Directly above the user stack is the number of arguments (argc), the argument vector (argv), and the process environment vector (envp) set up when the program was executed. Following them are the argument and environment strings themselves. Next is the signal code, used when the system delivers signals to the process. At the top is the ps_strings structure, used by ps to locate the argv of the process.

Historically most executables were statically linked. In a statically linked binary, all the library routines and system call entry stubs are loaded into the binary image at the time that it is compiled. Today, most binaries are dynamically linked. A dynamically linked binary contains only the compiled application code and a list of the routines (library and system call entry stubs) that it needs. When the executable is run, a set of shared libraries containing the routines that it needs to use are mapped into its address space as part of its startup. The first time that it calls a routine, that routine is located in the shared library and a dynamic linkage is created to it.

When the dynamic loader does the mmap system call to allocate space for the shared libraries, the kernel must find a place within the process address space to place them. The convention in FreeBSD is to place them just below the administrative lower limit for the stack. Since the stack will not be permitted to grow below the administrative stack size limit, there is no danger of the shared libraries being overwritten. A side effect of this implementation is that the stack limit cannot be safely changed after a binary begins running. Ideally, a bigger stack limit can be set by the process (such as the shell) before it starts the application. However, applications that know at startup that they will need a bigger stack can increase their stack limit and then call the exec system call on themselves to restart themselves with their shared libraries relocated at the bottom of their new stack limit.

An alternative would be to place the shared libraries just above the heap limit. However, this would mean that the heap limit could not be increased once the binary began running. As applications much more frequently want to increase their heap size than their stack size, the stack limit was selected as the appropriate location to place the shared libraries.

A process requires the use of some global system resources. The kernel maintains a linked list of processes that has an entry for each process in the system. Among other data, the process entries record information on scheduling and on virtual-memory allocation. Because the entire process address space, including the kernel stack for the process, may be swapped out of main memory, the process entry must record enough information to be able to locate the process and to bring that process back into memory. In addition, information needed while the process is swapped out (e.g., scheduling information) must be maintained in the process entry rather than in the user structure to avoid the kernel swapping in the process only to decide that it is not at a high enough priority to be run.

Other global resources associated with a process include space to record information about descriptors and page tables that record information about physical-memory utilization.


3.6. Timing Services

The kernel provides several different timing services to processes. These services include timers that run in real time and timers that run only while a process is executing.

Real Time

The system's time offset since January 1, 1970, Universal Coordinated Time (UTC), also known as the Epoch, is returned by the system call gettimeofday. Most modern processors (including the PC processors) maintain a battery-backup time-of-day register. This clock continues to run even if the processor is turned off. When the system boots, it consults the processor's time-of-day register to find out the current time. The system's time is then maintained by the clock interrupts. At each interrupt, the system increments its global time variable by an amount equal to the number of microseconds per tick. For the PC, running at 100 ticks per second, each tick represents 10,000 microseconds.

External Representation

Time is always exported from the system as microseconds, rather than as clock ticks, to provide a resolution-independent format. Internally, the kernel is free to select whatever tick rate best trades off clock-interrupt-handling overhead with timer resolution. As the tick rate per second increases, the resolution of the system timers improves, but the time spent dealing with hardclock interrupts increases. As processors become faster, the tick rate can be increased to provide finer resolution without adversely affecting user applications. Systems with real-time constraints often run the clock at 500 or 1000 ticks per second.

All filesystem (and other) timestamps are maintained in UTC offsets from the Epoch. Conversion to local time, including adjustment for daylight saving time, is handled externally to the system in the C library.

Adjustment of the Time

Often, it is desirable to maintain the same time on all the machines on a network. It is also possible to keep more accurate time than that available from the basic processor clock. For example, hardware is readily available that listens to the set of radio stations that broadcast UTC synchronization signals in the United States. When processes on different machines agree on a common time, they will wish to change the clock on their host processor to agree with the networkwide time value. One possibility is to change the system time to the network time using the settimeofday system call. Unfortunately, the settimeofday system call will result in time running backward on machines whose clocks were fast. Time running backward can confuse user programs (such as make) that expect time to invariably increase. To avoid this problem, the system provides the adjtime system call [Mills, 1992]. The adjtime system call takes a time delta (either positive or negative) and changes the rate at which time advances by 10 percent, faster or slower, until the time has been corrected. The operating system does the speedup by incrementing the global time by 11,000 microseconds for each tick and does the slowdown by incrementing the global time by 9,000 microseconds for each tick. Regardless, time increases monotonically, and user processes depending on the ordering of file-modification times are not affected. However, time changes that take tens of seconds to adjust will affect programs that are measuring time intervals by using repeated calls to gettimeofday.

Interval Time

The system provides each process with three interval timers. The real timer decrements in real time. An example of use for this timer is a library routine maintaining a wakeup-service queue. A SIGALRM signal is delivered to the process when this timer expires. The real-time timer is run from the timeout queue maintained by the softclock() routine (see Section 3.4).

The profiling timer decrements both in process virtual time (when running in user mode) and when the system is running on behalf of the process. It is designed to be used by processes to profile their execution statistically. A SIGPROF signal is delivered to the process when this timer expires. Each time that profclock() runs, it checks to see whether the currently running process has requested a profiling timer; if it has, profclock() decrements the timer and sends the process a signal when zero is reached.

The virtual timer decrements in process virtual time. It runs only when the process is executing in user mode. A SIGVTALRM signal is delivered to the process when this timer expires. The virtual timer is also implemented in profclock() as the profiling timer is, except that it decrements the timer for the current process only if it is executing in user mode and not if it is running in the kernel.


3.7. User, Group, and Other Identifiers

One important responsibility of an operating system is to implement access-control mechanisms. Most of these access-control mechanisms are based on the notions of individual users and of groups of users. Users are named by a 32-bit number called a user identifier (UID). UIDs are not assigned by the kernel but by an outside administrative authority. UIDs are the basis for accounting, for restricting access to privileged kernel operations (such as the request used to reboot a running system), for deciding to what processes a signal may be sent, and as a basis for filesystem access and disk-space allocation. A single user, termed the superuser (usually given the user name root), is trusted by the system and is permitted to do any supported kernel operation. The superuser is identified not by any specific name, such as root, but instead by a UID of zero.

Users are organized into groups. Groups are named by a 32-bit number called a group identifier (GID). GIDs, like UIDs, are used in the filesystem access-control facilities and in disk-space allocation.

The state of every FreeBSD process includes a UID and a set of GIDs. A process's filesystem-access privileges are defined by the UID and GIDs of the process (for the filesystem hierarchy beginning at the process's root directory). Normally, these identifiers are inherited automatically from the parent process when a new process is created. Only the superuser is permitted to alter the real UID or real GID of a process. This scheme enforces a strict compartmentalization of privileges and ensures that no user other than the superuser can gain privileges.

Each file has three sets of permission bits, for read, write, or execute permission for each of owner, group, and other. These permission bits are checked in the following order:

  1. If the UID of the file is the same as the UID of the process, only the owner permissions apply; the group and other permissions are not checked.

  2. If the UIDs do not match, but the GID of the file matches one of the GIDs of the process, only the group permissions apply; the owner and other permissions are not checked.

  3. Only if the UID and GIDs of the process fail to match those of the file are the permissions for all others checked. If these permissions do not allow the requested operation, it will fail.

The UID and GIDs for a process are inherited from its parent. When a user logs in, the login program (see Section 14.5) sets the UID and GIDs before doing the exec system call to run the user's login shell; thus, all subsequent processes will inherit the appropriate identifiers.

Often, it is desirable to grant a user limited additional privileges. For example, a user who wants to send mail must be able to append the mail to another user's mailbox. Making the target mailbox writable by all users would permit a user other than its owner to modify messages in it (whether maliciously or unintentionally). To solve this problem, the kernel allows the creation of programs that are granted additional privileges while they are running. Programs that run with a different UID are called set-user-identifier (setuid) programs; programs that run with an additional group privilege are called set-group-identifier (setgid) programs [Ritchie, 1979]. When a setuid program is executed, the permissions of the process are augmented to include those of the UID associated with the program. The UID of the program is termed the effective UID of the process, whereas the original UID of the process is termed the real UID. Similarly, executing a setgid program augments a process's permissions with those of the program's GID, and the effective GID and real GID are defined accordingly.

Systems can use setuid and setgid programs to provide controlled access to files or services. For example, the program that adds mail to the users' mailbox runs with the privileges of the superuser, which allow it to write to any file in the system. Thus, users do not need permission to write other users' mailboxes, but can still do so by running this program. Naturally, such programs must be written carefully to have only a limited set of functionality!

The UID and GIDs are maintained as part of the per-process state. Historically, GIDs were implemented as one distinguished GID (the effective GID) and a supplementary array of GIDs, which was logically treated as one set of GIDs. In FreeBSD, the distinguished GID is the first entry in the array of GIDs. The supplementary array is of a fixed size (16 in FreeBSD), but may be changed by recompiling the kernel.

FreeBSD implements the setgid capability by setting the zeroth element of the supplementary groups array of the process that executed the setgid program to the group of the file. Permissions can then be checked as it is for a normal process. Because of the additional group, the setgid program may be able to access more files than can a user process that runs a program without the special privilege. To avoid losing the privileges associated with the group in the zeroth array element when running a setgid program, the login program duplicates the zeroth array element into the first array element when initializing the user's supplementary group array. Thus, when a setgid program is run and modifies the zeroth element, the user does not lose any privileges as the group that had been contained in the zeroth array element is still available in the first array element.

The setuid capability is implemented by the effective UID of the process being changed from that of the user to that of the program being executed. As it will with setgid, the protection mechanism will now permit access without any change or special knowledge that the program is running setuid. Since a process can have only a single UID at a time, it is possible to lose some privileges while running setuid. The previous real UID is still maintained as the real UID when the new effective UID is installed. The real UID, however, is not used for any validation checking.

A setuid process may wish to revoke its special privilege temporarily while it is running. For example, it may need its special privilege to access a restricted file at only the start and end of its execution. During the rest of its execution, it should have only the real user's privileges. In earlier versions of BSD, revocation of privilege was done by switching of the real and effective UIDs. Since only the effective UID is used for access control, this approach provided the desired semantics and provided a place to hide the special privilege. The drawback to this approach was that the real and effective UIDs could easily become confused.

In FreeBSD, an additional identifier, the saved UID, is used to record the identity of setuid programs. When a program is exec'ed, its effective UID is copied to its saved UID. The first line of Table 3.2 shows an unprivileged program for which the real, effective, and saved UIDs are all those of the real user. The second line of Table 3.2 shows a setuid program being run that causes the effective UID to be set to its associated special-privilege UID. The special-privilege UID has also been copied to the saved UID.

Table 3.2. Actions affecting the real, effective, and saved UIDs.

Action

Real

Effective

Saved

1. exec-normal

R

R

R

2. exec-setuid

R

S

S

3. seteuid(R)

R

R

S

4. seteuid(S)

R

S

S

5. seteuid(R)

R

R

S

6. exec-normal

R

R

R

Key: R—real user identifier; S—special-privilege user identifier.

The seteuid system call sets only the effective UID; it does not affect the real or saved UIDs. The seteuid system call is permitted to set the effective UID to the value of either the real or the saved UID. Lines 3 and 4 of Table 3.2 show how a setuid program can give up and then reclaim its special privilege while continuously retaining its correct real UID. Lines 5 and 6 show how a setuid program can run a subprocess without granting the latter the special privilege. First, it sets its effective UID to the real UID. Then, when it exec's the subprocess, the effective UID is copied to the saved UID, and all access to the special-privilege UID is lost.

A similar saved GID mechanism permits processes to switch between the real GID and the initial effective GID.

Host Identifiers

An additional identifier is defined by the kernel for use on machines operating in a networked environment. A string (of up to 256 characters) specifying the host's name is maintained by the kernel. This value is intended to be defined uniquely for each machine in a network. In addition, in the Internet domain-name system, each machine is given a unique 32-bit number. Use of these identifiers permits applications to use networkwide unique identifiers for objects such as processes, files, and users, which is useful in the construction of distributed applications [Gifford, 1981]. The host identifiers for a machine are administered outside the kernel.

Process Groups and Sessions

Each process in the system is associated with a process group. The group of processes in a process group is sometimes referred to as a job and is manipulated as a single entity by processes such as the shell. Some signals (e.g., SIGINT) are delivered to all members of a process group, causing the group as a whole to suspend or resume execution, or to be interrupted or terminated.

Sessions were designed by the IEEE POSIX.1003.1 Working Group with the intent of fixing a long-standing security problem in UNIX—namely, that processes could modify the state of terminals that were trusted by another user's processes. A session is a collection of process groups, and all members of a process group are members of the same session. In FreeBSD, when a user first logs onto the system, he is entered into a new session. Each session has a controlling process, which is normally the user's login shell. All subsequent processes created by the user are part of process groups within this session, unless he explicitly creates a new session. Each session also has an associated login name, which is usually the user's login name. This name can be changed by only the superuser.

Each session is associated with a terminal, known as its controlling terminal. Each controlling terminal has a process group associated with it. Normally, only processes that are in the terminal's current process group read from or write to the terminal, allowing arbitration of a terminal between several different jobs. When the controlling process exits, access to the terminal is taken away from any remaining processes within the session.

Newly created processes are assigned process IDs distinct from all already-existing processes and process groups, and are placed in the same process group and session as their parent. Any process may set its process group equal to its process ID (thus creating a new process group) or to the value of any process group within its session. In addition, any process may create a new session, as long as it is not already a process-group leader. Sessions, process groups, and associated topics are discussed further in Section 4.8 and in Section 10.5.


3.8. Resource Services

All systems have limits imposed by their hardware architecture and configuration to ensure reasonable operation and to keep users from accidentally (or maliciously) creating resource shortages. At a minimum, the hardware limits must be imposed on processes that run on the system. It is usually desirable to limit processes further, below these hardware-imposed limits. The system measures resource utilization and allows limits to be imposed on consumption either at or below the hardware-imposed limits.

Process Priorities

The default scheduling policy in the FreeBSD system is managed by the share scheduler that gives CPU scheduling priority to processes that have not used CPU time recently. This priority scheme tends to favor processes that execute for only short periods of time—for example, interactive processes. The priority selected for each process is maintained internally by the kernel. The calculation of the priority is affected by the per-process nice variable. Positive nice values mean that the process is willing to receive less than its share of the processor. Negative values of nice mean that the process wants more than its share of the processor. Most processes run with the default nice value of zero, asking neither higher nor lower access to the processor. It is possible to determine or change the nice currently assigned to a process, to a process group, or to the processes of a specified user. Many factors other than nice affect scheduling, including the amount of CPU time that the process has used recently, the amount of memory that the process has used recently, and the current load on the system.

In addition to the share scheduler described here, the FreeBSD system also has a real-time scheduler available. The real-time scheduler allows processes to precisely control their order of execution and the amount of time given to each process. The details of the share and real-time scheduling algorithms are described in Section 4.4.

Resource Utilization

As a process executes, it uses system resources such as the CPU and memory. The kernel tracks the resources used by each process and compiles statistics describing this usage. The statistics managed by the kernel are available to a process while the latter is executing. When a process terminates, the statistics are made available to its parent via the wait family of system calls.

The resources used by a process are returned by the system call getrusage. The resources used by the current process, or by all the terminated children of the current process, may be requested. The following information is included:

  • The amount of user and system time used by the process

  • The memory utilization of the process

  • The paging and disk I/O activity of the process

  • The number of voluntary and involuntary context switches taken by the process

  • The amount of interprocess communication done by the process

The resource-usage information is collected at locations throughout the kernel. The CPU time is collected by the statclock() function, which is called either by the system clock in hardclock(), or, if an alternate clock is available, by the alternate-clock interrupt process. The kernel scheduler calculates memory utilization by sampling the amount of memory that an active process is using at the same time that it is recomputing process priorities. The vm_fault() routine recalculates the paging activity each time that it starts a disk transfer to fulfill a paging request (see Section 5.11). The I/O activity statistics are collected each time that the process has to start a transfer to fulfill a file or device I/O request, as well as when the general system statistics are calculated. The IPC communication activity is updated each time that information is sent or received.

Resource Limits

The kernel also supports limiting of certain per-process resources. These resources include the following:

  • The maximum amount of CPU time that can be accumulated

  • The maximum bytes that a process can request be locked into memory

  • The maximum size of a process's data segment

  • The maximum size of a process's stack segment

  • The maximum amount of private physical memory that a process may have at any given moment

  • The maximum amount of private or shared physical memory that a process may have at any given moment

  • The maximum amount of physical memory that a process may have dedicated to socket buffers

  • The maximum size of a file that can be created by a process

  • The maximum size of a core file that can be created by a process

  • The maximum number of simultaneous open files for a process

  • The maximum number of simultaneous processes allowed to a user

For each resource controlled by the kernel, two limits are maintained: a soft limit and a hard limit. All users can alter the soft limit within the range of 0 to the corresponding hard limit. All users can (irreversibly) lower the hard limit, but only the superuser can raise the hard limit. If a process exceeds certain soft limits, a signal is delivered to the process to notify it that a resource limit has been exceeded. Normally, this signal causes the process to terminate, but the process may either catch or ignore the signal. If the process ignores the signal and fails to release resources that it already holds, further attempts to obtain more resources will result in errors.

Resource limits are generally enforced at or near the locations that the resource statistics are collected. The CPU time limit is enforced in the process context-switching function. The stack and data-segment limits are enforced by a return of allocation failure once those limits have been reached. The file-size limit is enforced by the filesystem.

Filesystem Quotas

In addition to limits on the size of individual files, the kernel optionally enforces limits on the total amount of space that a user or group can use on a filesystem. Our discussion of the implementation of these limits is deferred to Section 7.4.


3.9. System-Operation Services

There are several operational functions having to do with system startup and shutdown. The bootstrapping operations are described in Section 14.2. System shutdown is described in Section 14.6.

Accounting

The system supports a simple form of resource accounting. As each process terminates, an accounting record describing the resources used by that process is written to a systemwide accounting file. The following information is supplied by the system:

  • The name of the command that ran

  • The amount of user and system CPU time that was used

  • The time the command started

  • The elapsed time the command ran

  • The average amount of memory used

  • The number of disk I/O operations done

  • The UID and GID of the process

  • The terminal from which the process was started

  • Flags that tell whether the process fork'ed but did not exec, used superuser privileges, ran in compatibility mode, dumped core, and/or was killed by a signal.

The information in the accounting record is drawn from the run-time statistics that were described in Section 3.8. The granularity of the time fields is in sixty-fourths of a second. To conserve space in the accounting file, the times are stored in a 16-bit word as a floating-point number using 3 bits as a base-8 exponent, and the other 13 bits as the fractional part. For historic reasons, the same floating-point-conversion routine processes the count of disk operations, so the number of disk operations must be multiplied by 64 before it is converted to the floating-point representation.

The superuser requests accounting by passing the name of the file to be used for accounting to the kernel. As part of a process exiting, the kernel appends an accounting record to the accounting file. The kernel makes no use of the accounting records; the records' summaries and use are entirely the domain of user-level accounting programs. As a guard against a filesystem running out of space because of unchecked growth of the accounting file, the system suspends accounting when the filesystem is reduced to only 2 percent remaining free space. Accounting resumes when the filesystem has at least 4 percent free space.

The accounting information has certain limitations. The information on runtime and memory usage is only approximate because it is gathered statistically. Accounting information is written only when a process exits, so processes that are still running when a system is shut down unexpectedly do not show up in the accounting file. (Obviously, long-lived system daemons are among such processes.) Finally, the accounting records fail to include much information needed to do accurate billing, including usage of other resources, such as tape drives and printers.


Exercises

3.1 Describe three types of system activity.

3.2 When can a routine executing in the top half of the kernel be preempted? When can it be interrupted?

3.3 Why are routines executing in the bottom half of the kernel precluded from using information located in the current user process?

3.4 Why does the system defer as much work as possible from high-priority interrupts to lower-priority software-interrupt processes?

3.5 What determines the shortest (nonzero) time period that a user process can request when setting a timer?

3.6 How does the kernel determine the system call for which it has been invoked?

3.7 How are initialized data represented in an executable file? How are uninitialized data represented in an executable file? Why are the representations different?

3.8 Describe how the "#!" mechanism can be used to make programs that require emulation appear as though they were normal executables.

3.9 Is it possible for a file to have permissions set such that its owner cannot read it, even though a group can? Is this situation possible if the owner is a member of the group that can read the file? Explain your answers.

*3.10 Describe the security implications of not zero filling the stack region at program startup.

*3.11 Why is the conversion from UTC to local time done by user processes rather than in the kernel?

*3.12 What is the advantage of having the kernel rather than an application restart an interrupted system call?

*3.13 Describe a scenario in which the timer-wheel algorithm used for the callout queue does not work well. Suggest an alternative data structure that runs more quickly than does the timer-wheel algorithm for your scenario.

*3.14 The SIGPROF profiling timer was originally intended to replace the profil system call to collect a statistical sampling of a program's program counter. Give two reasons why the profil facility had to be retained.

**3.15 What weakness in the process-accounting mechanism makes the latter unsuitable for use in a commercial environment?


References

Barkley & Lee, 1988. R. E. Barkley & T. P. Lee, "A Heap-Based Callout Implementation to Meet Real-Time Needs," USENIX Association Conference Proceedings, pp. 213-222, June 1988.

Gifford, 1981. D. Gifford, "Information Storage in a Decentralized Computer System," PhD Thesis, Electrical Engineering Department, Stanford University, Stanford, CA, 1981.

McCanne & Torek, 1993. S. McCanne & C. Torek, "A Randomized Sampling Clock for CPU Utilization Estimation and Code Profiling," USENIX Association Conference Proceedings, pp. 387-394, January 1993.

Mills, 1992. D. L. Mills, "The NTP Time Synchronization Protocol," RFC 1305, available from http://www.faqs.org/rfcs/rfcl305.html, March 1992.

Ritchie, 1979. D. M. Ritchie, "Protection of Data File Contents," United States Patent, no. 4,135,240, United States Patent Office, Washington, DC, January 16, 1979. Assignee: Bell Telephone Laboratories, Inc., Murray Hill, NJ, Appl. No.: 377,591, Filed: July 9, 1973.

Varghese & Lauck, 1987. G. Varghese & T. Lauck, "Hashed and Hierarchical Timing Wheels: Data Structures for the Efficient Implementation of a Timer Facility," Proceedings of the Eleventh Symposium on Operating Systems Principles, pp. 25-38, November 1987.


Part II: Processes

    Chapter 4. Process Management

    Chapter 5. Memory Management


Chapter 4. Process Management


4.1. Introduction to Process Management

A process is a program in execution. A process must have system resources, such as memory and the underlying CPU. The kernel supports the illusion of concurrent execution of multiple processes by scheduling system resources among the set of processes that are ready to execute. On a multiprocessor, multiple processes may really execute concurrently. This chapter describes the composition of a process, the method that the system uses to switch between processes, and the scheduling policy that it uses to promote sharing of the CPU. It also introduces process creation and termination, and details the signal facilities and process-debugging facilities.

Two months after the developers began the first implementation of the UNIX operating system, there were two processes: one for each of the terminals of the PDP-7. At age 10 months, and still on the PDP-7, UNIX had many processes, the fork operation, and something like the wait system call. A process executed a new program by reading in a new program on top of itself. The first PDP-11 system (First Edition UNIX) saw the introduction of exec. All these systems allowed only one process in memory at a time. When a PDP-11 with memory management (a KS-11) was obtained, the system was changed to permit several processes to remain in memory simultaneously, to reduce swapping. But this change did not apply to multiprogramming because disk I/O was synchronous. This state of affairs persisted into 1972 and the first PDP-11/45 system. True multiprogramming was finally introduced when the system was rewritten in C. Disk I/O for one process could then proceed while another process ran. The basic structure of process management in UNIX has not changed since that time [Ritchie, 1988].

A process operates in either user mode or kernel mode. In user mode, a process executes application code with the machine in a nonprivileged protection mode. When a process requests services from the operating system with a system call, it switches into the machine's privileged protection mode via a protected mechanism and then operates in kernel mode.

The resources used by a process are similarly split into two parts. The resources needed for execution in user mode are defined by the CPU architecture and typically include the CPU's general-purpose registers, the program counter, the processor-status register, and the stack-related registers, as well as the contents of the memory segments that constitute the FreeBSD notion of a program (the text, data, shared library, and stack segments).

Kernel-mode resources include those required by the underlying hardware— such as registers, program counter, and stack pointer—and also by the state required for the FreeBSD kernel to provide system services for a process. This kernel state includes parameters to the current system call, the current process's user identity, scheduling information, and so on. As described in Section 3.1, the kernel state for each process is divided into several separate data structures, with two primary structures: the process structure and the user structure.

The process structure contains information that must always remain resident in main memory, along with references to other structures that remain resident, whereas the user structure contains information that needs to be resident only when the process is executing (although user structures of other processes also may be resident). User structures are allocated dynamically through the memory-management facilities. Historically, more than one-half of the process state was stored in the user structure. In FreeBSD, the user structure is used for only a couple of structures that are referenced from the process structure. Process structures are allocated dynamically as part of process creation and are freed as part of process exit.

Multiprogramming

The FreeBSD system supports transparent multiprogramming: the illusion of concurrent execution of multiple processes or programs. It does so by context switching—that is, by switching between the execution context of processes. A mechanism is also provided for scheduling the execution of processes—that is, for deciding which one to execute next. Facilities are provided for ensuring consistent access to data structures that are shared among processes.

Context switching is a hardware-dependent operation whose implementation is influenced by the underlying hardware facilities. Some architectures provide machine instructions that save and restore the hardware-execution context of the process, including the virtual-address space. On the others, the software must collect the hardware state from various registers and save it, then load those registers with the new hardware state. All architectures must save and restore the software state used by the kernel.

Context switching is done frequently, so increasing the speed of a context switch noticeably decreases time spent in the kernel and provides more time for execution of user applications. Since most of the work of a context switch is expended in saving and restoring the operating context of a process, reducing the amount of the information required for that context is an effective way to produce faster context switches.

Scheduling

Fair scheduling of processes is an involved task that is dependent on the types of executable programs and on the goals of the scheduling policy. Programs are characterized according to the amount of computation and the amount of I/O that they do. Scheduling policies typically attempt to balance resource utilization against the time that it takes for a program to complete. In FreeBSD's default scheduler, which we shall refer to as the time-share scheduler, a process's priority is periodically recalculated based on various parameters, such as the amount of CPU time it has used, the amount of memory resources it holds or requires for execution, and so on. Some tasks require more precise control over process execution called real-time scheduling, which must ensure that processes finish computing their results by a specified deadline or in a particular order. The FreeBSD kernel implements real-time scheduling with a separate queue from the queue used for regular time-shared processes. A process with a real-time priority is not subject to priority degradation and will only be preempted by another process of equal or higher real-time priority. The FreeBSD kernel also implements a queue of processes running at idle priority. A process with an idle priority will run only when no other process in either the real-time or time-share-scheduled queues is runnable and then only if its idle priority is equal or greater than all other runnable idle priority processes.

The FreeBSD time-share scheduler uses a priority-based scheduling policy that is biased to favor interactive programs, such as text editors, over long-running batch-type jobs. Interactive programs tend to exhibit short bursts of computation followed by periods of inactivity or I/O. The scheduling policy initially assigns to each process a high execution priority and allows that process to execute for a fixed time slice. Processes that execute for the duration of their slice have their priority lowered, whereas processes that give up the CPU (usually because they do I/O) are allowed to remain at their priority. Processes that are inactive have their priority raised. Thus, jobs that use large amounts of CPU time sink rapidly to a low priority, whereas interactive jobs that are mostly inactive remain at a high priority so that, when they are ready to run, they will preempt the long-running lower-priority jobs. An interactive job, such as a text editor searching for a string, may become compute-bound briefly and thus get a lower priority, but it will return to a high priority when it is inactive again while the user thinks about the result.

Some tasks such as the compilation of a large application may be done in many small steps in which each component is compiled in a separate process. No individual step runs long enough to have its priority degraded, so the compilation as a whole impacts the interactive programs. To detect and avoid this problem, the scheduling priority of a child process is propagated back to its parent. When a new child process is started, it begins running with its parents current priority. Thus, as the program that coordinates the compilation (typically make) starts many compilation steps, its priority is dropped because of the CPU-intensive behavior of its children. Later compilation steps started by make begin running and stay at a lower priority, which allows higher-priority interactive programs to run in preference to them as desired.

The system also needs a scheduling policy to deal with problems that arise from not having enough main memory to hold the execution contexts of all processes that want to execute. The major goal of this scheduling policy is to minimize thrashing—a phenomenon that occurs when memory is in such short supply that more time is spent in the system handling page faults and scheduling processes than in user mode executing application code.

The system must both detect and eliminate thrashing. It detects thrashing by observing the amount of free memory. When the system has few free memory pages and a high rate of new memory requests, it considers itself to be thrashing. The system reduces thrashing by marking the least-recently run process as not being allowed to run. This marking allows the pageout daemon to push all the pages associated with the process to backing store. On most architectures, the kernel also can push to backing store the user structure of the marked process. The effect of these actions is to cause the process to be swapped out (see Section 5.12). The memory freed by blocking the process can then be distributed to the remaining processes, which usually can then proceed. If the thrashing continues, additional processes are selected for being blocked from running until enough memory becomes available for the remaining processes to run effectively. Eventually, enough processes complete and free their memory that blocked processes can resume execution. However, even if there is not enough memory, the blocked processes are allowed to resume execution after about 20 seconds. Usually, the thrashing condition will return, requiring that some other process be selected for being blocked (or that an administrative action be taken to reduce the load).


4.2. Process State

Every process in the system is assigned a unique identifier termed the process identifier (PID). PIDs are the common mechanism used by applications and by the kernel to reference processes. PIDs are used by applications when the latter are sending a signal to a process and when receiving the exit status from a deceased process. Two PIDs are of special importance to each process: the PID of the process itself and the PID of the process's parent process.

The layout of process state was completely reorganized in FreeBSD 5.2. The goal was to support multiple threads that share an address space and other resources. Threads have also been called lightweight processes in other systems. A thread is the unit of execution of a process; it requires an address space and other resources, but it can share many of those resources with other threads. Threads sharing an address space and other resources are scheduled independently and can all do system calls simultaneously. The reorganization of process state in FreeBSD 5.2 was designed to support threads that can select the set of resources to be shared, known as variable-weight processes [Aral et al., 1989].

The developers did the reorganization by moving many components of process state from the process and user structures into separate substructures for each type of state information, as shown in Figure 4.1. The process structure references all the substructures directly or indirectly. The user structure remains primarily as a historic artifact for the benefit of debuggers. The thread structure contains just the information needed to run in the kernel: information about scheduling, a stack to use when running in the kernel, a thread control block (TCB), and other machine-dependent state. The TCB is defined by the machine architecture; it includes the general-purpose registers, stack pointers, program counter, processor-status longword, and memory-management registers.

Figure 4.1. Process state.

In their lightest-weight form, FreeBSD threads share all the process resources including the PID. When additional parallel computation is needed, a new thread is created using the kse_create system call. All the scheduling and management of the threads is handled by a user-level scheduler that is notified of thread transitions via callbacks from the kernel. The user-level thread manager must also keep track of the user-level stacks being used by each of the threads, since the entire address space is shared including the area normally used for the stack. Since the threads all share a single process structure, they have only a single PID and thus show up as a single entry in the ps listing.

Many applications do not wish to share all of a process's resources. The rfork system call creates a new process entry that shares a selected set of resources from its parent. Typically the signal actions, statistics, and the stack and data parts of the address space are not shared. Unlike the lightweight thread created by kse_create, the rfork system call associates a PID with each thread that shows up in a ps listing and that can be manipulated in the same way as any other process in the system. Processes created by fork, vfork, or rfork have just a single thread structure associated with them.

The Process Structure

In addition to the references to the substructures, the process entry shown in Figure 4.1 contains the following categories of information:

  • Process identification: the PID and the parent PID

  • Signal state: signals pending delivery, signal mask, and summary of signal actions

  • Tracing: process tracing information

  • Timers: real-time timer and CPU-utilization counters

The process substructures shown in Figure 4.1 have the following categories of information:

  • Process-group identification: the process group and the session to which the process belongs

  • User credentials: the real, effective, and saved user and group identifiers

  • Memory management: the structure that describes the allocation of virtual address space used by the process; the VM space and its related structures are described more fully in Chapter 5

  • File descriptors: an array of pointers to file entries indexed by the process open file descriptors; also, the open file flags and current directory

  • System call vector: The mapping of system call numbers to actions; in addition to current and deprecated native FreeBSD executable formats, the kernel can run binaries compiled for several other UNIX variants such as Linux, OSF/1, and System V Release 4 by providing alternative system call vectors when such environments are requested

  • Resource accounting: the rlimit structures that describe the utilization of the many resources provided by the system (see Section 3.8)

  • Statistics: statistics collected while the process is running that are reported when it exits and are written to the accounting file; also, includes process timers and profiling information if the latter is being collected

  • Signal actions: the action to take when a signal is posted to a process

  • Thread structure: The contents of the thread structure (described at the end of this section)

The state element of the process structure holds the current value of the process state. The possible state values are shown in Table 4.1. When a process is first created with a fork system call, it is initially marked as NEW. The state is changed to NORMAL when enough resources are allocated to the process for the latter to begin execution. From that point onward, a process's state will fluctuate among NORMAL (where its thread(s) will be either RUNNABLE—that is, preparing to be or actually executing; SLEEPING—that is, waiting for an event; or STOPPED—that is, stopped by a signal or the parent process) until the process terminates. A deceased process is marked as ZOMBIE until it has freed its resources and communicated its termination status to its parent process.

Table 4.1. Process states.

State

Description

NEW

undergoing process creation

NORMAL

thread(s) will be RUNNABLE, SLEEPING, or STOPPED

ZOMBIE

undergoing process termination

The system organizes process structures into two lists. Process entries are on the zombproc list if the process is in the ZOMBIE state; otherwise, they are on the allproc list. The two queues share the same linkage pointers in the process structure, since the lists are mutually exclusive. Segregating the dead processes from the live ones reduces the time spent both by the wait system call, which must scan the zombies for potential candidates to return, and by the scheduler and other functions that must scan all the potentially runnable processes.

Most threads, except the currently executing thread (or threads if the system is running on a multiprocessor), are also in one of two queues: a run queue or a sleep queue. Threads that are in a runnable state are placed on a run queue, whereas threads that are blocked awaiting an event are located on a sleep queue. Stopped threads awaiting an event are located on a sleep queue, or they are on neither type of queue. The run queues are organized according to thread-scheduling priority and are described in Section 4.4. The sleep queues are organized in a data structure that is hashed by event identifier. This organization optimizes finding the sleeping threads that need to be awakened when a wakeup occurs for an event. The sleep queues are described in Section 4.3.

The p_pptr pointer and related lists (p_children and p_sibling) are used in locating related processes, as shown in Figure 4.2 (on page 86). When a process spawns a child process, the child process is added to its parent's p_children list. The child process also keeps a backward link to its parent in its p_pptr pointer. If a process has more than one child process active at a time, the children are linked together through their p_sibling list entries. In Figure 4.2, process B is a direct descendant of process A, whereas processes C, D, and E are descendants of process B and are siblings of one another. Process B typically would be a shell that started a pipeline (see Sections 2.4 and 2.6) including processes C, D, and E. Process A probably would be the system-initialization process init (see Sections 3.1 and 14.6).

Figure 4.2. Process-group hierarchy.

CPU time is made available to threads according to their scheduling class and scheduling priority. As shown in Table 4.2, the FreeBSD kernel has two kernel and three user scheduling classes. The kernel will always run the thread in the highest-priority class. Any kernel-interrupt threads will run in preference to anything else followed by any top-half-kernel threads. Any runnable real-time threads are run in preference to runnable threads in the share and idle classes. Runnable time-share threads are run in preference to runnable threads in the idle class. The priorities of threads in the real-time and idle classes are set by the applications using the rtprio system call and are never adjusted by the kernel. The bottom-half interrupt priorities are set when the devices are configured and never change. The top-half priorities are set based on predefined priorities for each kernel subsystem and never change.

Table 4.2. Thread-scheduling classes.

Range

Class

Thread type

0–63

ITHD

Bottom-half kernel (interrupt)

64-127

KERN

Top-half kernel

128-159

REALTIME

Real-time user

160-223

TIMESHARE

Time-sharing user

224-255

IDLE

Idle user

The priorities of threads running in the time-share class are adjusted by the kernel based on resource usage and recent CPU utilization. A thread has two scheduling priorities: one for scheduling user-mode execution and one for scheduling kernel-mode execution. The kg_user_pri field associated with the thread structure contains the user-mode scheduling priority, whereas the td_priority field holds the current scheduling priority. The current priority may be different from the user-mode priority when the thread is executing in the top half of the kernel. Priorities range between 0 and 255, with a lower value interpreted as a higher priority (see Table 4.2). User-mode priorities range from 128 to 255; priorities less than 128 are used only when a thread is asleep—that is, awaiting an event in the kernel—and immediately after such a thread is awakened. Threads in the kernel are given a higher priority because they typically hold shared kernel resources when they awaken. The system wants to run them as quickly as possible once they get a resource so that they can use the resource and return it before another thread requests it and gets blocked waiting for it.

When a thread goes to sleep in the kernel, it must specify whether it should be awakened and marked runnable if a signal is posted to it. In FreeBSD, a kernel thread will be awakened by a signal only if it sets the PCATCH flag when it sleeps. The msleep() interface also handles sleeps limited to a maximum time duration and the processing of restartable system calls. The msleep() interface includes a reference to a string describing the event that the thread awaits; this string is externally visible—for example, in ps. The decision of whether to use an interruptible sleep depends on how long the thread may be blocked. Because it is complex to be prepared to handle signals in the midst of doing some other operation, many sleep requests are not interruptible; that is, a thread will not be scheduled to run until the event for which it is waiting occurs. For example, a thread waiting for disk I/O will sleep with signals blocked.

For quickly occurring events, delaying to handle a signal until after they complete is imperceptible. However, requests that may cause a thread to sleep for a long period, such as waiting for terminal or network input, must be prepared to have their sleep interrupted so that the posting of signals is not delayed indefinitely. Threads that sleep interruptibly may abort their system call because of a signal arriving before the event for which they are waiting has occurred. To avoid holding a kernel resource permanently, these threads must check why they have been awakened. If they were awakened because of a signal, they must release any resources that they hold. They must then return the error passed back to them by msleep(), which will be EINTR if the system call is to be aborted after the signal or ERESTART if it is to be restarted. Occasionally, an event that is supposed to occur quickly, such as a disk I/O, will get held up because of a hardware failure. Because the thread is sleeping in the kernel with signals blocked, it will be impervious to any attempts to send it a signal, even a signal that should cause it to exit unconditionally. The only solution to this problem is to change sleep()s on hardware events that may hang to be interruptible.

In the remainder of this book, we shall always use sleep() when referring to the routine that puts a thread to sleep, even when the msleep() interface is the one that is being used.

The Thread Structure

The thread structure shown in Figure 4.1 contains the following categories of information:

  • Scheduling: the thread priority, user-mode scheduling priority, recent CPU utilization, and amount of time spent sleeping

  • Thread state: the run state of a thread (runnable, sleeping); additional status flags; if the thread is sleeping, the wait channel, the identity of the event for which the thread is waiting (see Section 4.3), and a pointer to a string describing the event

  • Machine state: the machine-dependent thread information

  • TCB: the user- and kernel-mode execution states

  • Kernel stack: the per-thread execution stack for the kernel

Historically, the kernel stack was mapped to a fixed location in the virtual address space. The reason for using a fixed mapping is that when a parent forks, its runtime stack is copied for its child. If the kernel stack is mapped to a fixed address, the child's kernel stack is mapped to the same addresses as its parent kernel stack. Thus, all its internal references, such as frame pointers and stack-variable references, work as expected.

On modern architectures with virtual address caches, mapping the user structure to a fixed address is slow and inconvenient. FreeBSD 5.2 removes this constraint by eliminating all but the top call frame from the child's stack after copying it from its parent so that it returns directly to user mode, thus avoiding stack copying and relocation problems.

Every thread that might potentially run must have its stack resident in memory because one task of its stack is to handle page faults. If it were not resident, it would page fault when the thread tried to run, and there would be no kernel stack available to service the page fault. Since a system may have many thousands of threads, the kernel stacks must be kept small to avoid wasting too much physical memory. In FreeBSD 5.2 on the PC, the kernel stack is limited to two pages of memory. Implementors must be careful when writing code that executes in the kernel to avoid using large local variables and deeply nested subroutine calls, to avoid overflowing the run-time stack. As a safety precaution, some architectures leave an invalid page between the area for the run-time stack and the data structures that follow it. Thus, overflowing the kernel stack will cause a kernel-access fault instead of disastrously overwriting other data structures. It would be possible to simply kill the process that caused the fault and continue running. However, the FreeBSD kernel panics on a kernel-access fault because such a fault shows a fundamental design error in the kernel. By panicking and creating a crash dump, the error can usually be pinpointed and corrected.


4.3. Context Switching

The kernel switches among threads in an effort to share the CPU effectively; this activity is called context switching. When a thread executes for the duration of its time slice or when it blocks because it requires a resource that is currently unavailable, the kernel finds another thread to run and context switches to it. The system can also interrupt the currently executing thread to run a thread triggered by an asynchronous event, such as a device interrupt. Although both scenarios involve switching the execution context of the CPU, switching between threads occurs synchronously with respect to the currently executing thread, whereas servicing interrupts occurs asynchronously with respect to the current thread. In addition, interprocess context switches are classified as voluntary or involuntary. A voluntary context switch occurs when a thread blocks because it requires a resource that is unavailable. An involuntary context switch takes place when a thread executes for the duration of its time slice or when the system identifies a higher-priority thread to run.

Each type of context switching is done through a different interface. Voluntary context switching is initiated with a call to the sleep() routine, whereas an involuntary context switch is forced by direct invocation of the low-level context-switching mechanism embodied in the mi_switch() and setrunnable() routines. Asynchronous event handling is triggered by the underlying hardware and is effectively transparent to the system. Our discussion will focus on how asynchronous event handling relates to synchronizing access to kernel data structures.

Thread State

Context switching between threads requires that both the kernel- and user-mode context be changed. To simplify this change, the system ensures that all a thread's user-mode state is located in one data structure: the thread structure (most kernel state is kept elsewhere). The following conventions apply to this localization:

  • Kernel-mode hardware-execution state: Context switching can take place in only kernel mode. The kernel's hardware-execution state is defined by the contents of the TCB that is located in the thread structure.

  • User-mode hardware-execution state: When execution is in kernel mode, the user-mode state of a thread (such as copies of the program counter, stack pointer, and general registers) always resides on the kernel's execution stack that is located in the thread structure. The kernel ensures this location of user-mode state by requiring that the system-call and trap handlers save the contents of the user-mode execution context each time that the kernel is entered (see Section 3.1).

  • The process structure: The process structure always remains resident in memory.

  • Memory resources: Memory resources of a process are effectively described by the contents of the memory-management registers located in the TCB and by the values present in the process and thread structures. As long as the process remains in memory, these values will remain valid, and context switches can be done without the associated page tables being saved and restored. However, these values need to be recalculated when the process returns to main memory after being swapped to secondary storage.

Low-Level Context Switching

The localization of the context of a process in the latter's thread structure permits the kernel to do context switching simply by changing the notion of the current thread structure and (if necessary) process structure, and restoring the context described by the TCB within the thread structure (including the mapping of the virtual address space). Whenever a context switch is required, a call to the mi_switch() routine causes the highest-priority thread to run. The mi_switch() routine first selects the appropriate thread from the scheduling queues, and then resumes the selected thread by loading its process's context from its TCB. Once mi_switch() has loaded the execution state of the new thread, it must also check the state of the new thread for a nonlocal return request (such as when a process first starts execution after a fork; see Section 4.5).

Voluntary Context Switching

A voluntary context switch occurs whenever a thread must await the availability of a resource or the arrival of an event. Voluntary context switches happen frequently in normal system operation. For example, a thread typically blocks each time that it requests data from an input device, such as a terminal or a disk. In FreeBSD, voluntary context switches are initiated through the sleep() routine. When a thread no longer needs the CPU, it invokes sleep() with a scheduling priority and a wait channel. The priority specified in a sleep() call is the priority that should be assigned to the thread when that thread is awakened. This priority does not affect the user-level scheduling priority.

The wait channel is typically the address of some data structure that identifies the resource or event for which the thread is waiting. For example, the address of a disk buffer is used while the thread is waiting for the buffer to be filled. When the buffer is filled, threads sleeping on that wait channel will be awakened. In addition to the resource addresses that are used as wait channels, there are some addresses that are used for special purposes:

  • The global variable lbolt is awakened by the scheduler once per second. Threads that want to wait for up to 1 second can sleep on this global variable. For example, the terminal-output routines sleep on lbolt while waiting for output-queue space to become available. Because queue space rarely runs out, it is easier simply to check for queue space once per second during the brief periods of shortages than it is to set up a notification mechanism such as that used for managing disk buffers. Programmers can also use the lbolt wait channel as a crude watchdog timer when doing debugging.

  • When a parent process does a wait system call to collect the termination status of its children, it must wait for one of those children to exit. Since it cannot know which of its children will exit first, and since it can sleep on only a single wait channel, there is a quandary on how to wait for the next of multiple events. The solution is to have the parent sleep on its own process structure. When a child exits, it awakens its parent's process-structure address rather than its own. Thus, the parent doing the wait will awaken independent of which child process is the first to exit. Once running, it must scan its list of children to determine which one exited.

  • When a thread does a sigpause system call, it does not want to run until it receives a signal. Thus, it needs to do an interruptible sleep on a wait channel that will never be awakened. By convention, the address of the user structure is given as the wait channel.

Sleeping threads are organized in an array of queues (see Figure 4.3). The sleep() and wakeup() routines hash wait channels to calculate an index into the sleep queues. The sleep() routine takes the following steps in its operation:

1. Prevent interrupts that might cause thread-state transitions by acquiring the sched_lock mutex (mutexes are explained in the next section).

2. Record the wait channel in the thread structure and hash the wait-channel value to locate a sleep queue for the thread.

3. Set the thread's priority to the priority that the thread will have when the thread is awakened and set the SLEEPING flag.

4. Place the thread at the end of the sleep queue selected in step 2.

5. Call mi_switch() to request that a new thread be scheduled; the sched_lock mutex is released as part of switching to the other thread.

Figure 4.3. Queueing structure for sleeping threads.

[View full size image]

A sleeping thread is not selected to execute until it is removed from a sleep queue and is marked runnable. This operation is done by the wakeup() routine, which is called to signal that an event has occurred or that a resource is available. Wakeup() is invoked with a wait channel, and it awakens all threads sleeping on that wait channel. All threads waiting for the resource are awakened to ensure that none are inadvertently left sleeping. If only one thread were awakened, it might not request the resource on which it was sleeping, and any other threads waiting for that resource would be left sleeping forever. A thread that needs an empty disk buffer in which to write data is an example of a thread that may not request the resource on which it was sleeping. Such a thread can use any available buffer. If none is available, it will try to create one by requesting that a dirty buffer be written to disk and then waiting for the I/O to complete. When the I/O finishes, the thread will awaken and will check for an empty buffer. If several are available, it may not use the one that it cleaned, leaving any other threads waiting for the buffer that it cleaned sleeping forever.

In instances where a thread will always use a resource when it becomes available, wakeup_one() can be used instead of wakeup(). The wakeup_one() routine wakes up only the first thread that it finds waiting for a resource. The assumption is that when the awakened thread is done with the resource it will issue another wakeup_one() to notify the next waiting thread that the resource is available. The succession of wakeup_one() calls will continue until all threads waiting for the resource have been awakened and had a chance to use it.

To avoid having excessive numbers of threads awakened, kernel programmers try to use wait channels with fine enough granularity that unrelated uses will not collide on the same resource. Thus, they put locks on each buffer in the buffer cache rather than putting a single lock on the buffer cache as a whole. The problem of many threads awakening for a single resource is further mitigated on a uniprocessor by the latter's inherently single-threaded operation. Although many threads will be put into the run queue at once, only one at a time can execute. Since the uniprocessor kernel runs nonpreemptively, each thread will run its system call to completion before the next one will get a chance to execute. Unless the previous user of the resource blocked in the kernel while trying to use the resource, each thread waiting for the resource will be able to get and use the resource when it is next run.

A wakeup() operation processes entries on a sleep queue from front to back. For each thread that needs to be awakened, wakeup() does the following:

  1. Removes the thread from the sleep queue

  2. Recomputes the user-mode scheduling priority if the thread has been sleeping longer than one second

  3. Makes the thread runnable if it is in a SLEEPING state and places the thread on the run queue if its process is not swapped out of main memory. If the process has been swapped out, the swapin process will be awakened to load it back into memory (see Section 5.12); if the thread is in a STOPPED state, it is not put on a run queue until it is explicitly restarted by a user-level process, either by a ptrace system call (see Section 4.9) or by a continue signal (see Section 4.7)

If wakeup() moved any threads to the run queue and one of them had a scheduling priority higher than that of the currently executing thread, it will also request that the CPU be rescheduled as soon as possible.

Synchronization

Historically, BSD systems ran only on uniprocessor architectures. Once a process began running in the top half of the kernel, it would run to completion or until it needed to sleep waiting for a resource to become available. The only contention for data structure access occurred at the points at which it slept or during an interrupt that needed to update a shared data structure. Synchronization with other processes was handled by ensuring that all shared data structures were consistent before going to sleep. Synchronization with interrupts was handled by raising the processor priority level to guard against interrupt activity while the shared data structure was manipulated.

Simple multiprocessor support was added to FreeBSD 3.0. It worked by creating a giant lock that protected the kernel. When a process entered the kernel it would have to acquire the giant lock before it could proceed. Thus, at most one processor could run in the kernel at a time. All the other processors could run only processes executing in user mode.

Symmetric multiprocessing (SMP) first appeared in FreeBSD 5.0 and required the addition of new synchronization schemes to eliminate the uniprocessor assumptions implicit in the FreeBSD 4.0 kernel [Schimmel, 1994]. Some subsystems in the FreeBSD 5.2 kernel have not yet been converted to symmetric multiprocessing and are still protected by the giant lock. However, most of the heavily used parts of the kernel have been moved out from under the giant lock, including much of the virtual memory system, the networking stack, and the filesystem.

Table 4.3 shows the hierarchy of locking that is necessary to support symmetric multiprocessing. The column labeled sleep in Table 4.3 shows whether a lock of that type may be held when a thread goes to sleep. At the lowest level, the hardware must provide a memory interlocked test-and-set instruction. The test-and-set instruction must allow two operations to be done on a main-memory location—the reading of the existing value followed by the writing of a new value—without any other processor being able to read or write that memory location between the two memory operations. Some architectures support more complex versions of the test-and-set instruction. For example, the PC provides a memory interlocked compare-and-swap instruction.

Table 4.3. Locking hierarchy.

Level

Type

Sleep

Description

Lowest

hardware

no

memory-interlocked test-and-set

spin mutex

no

spin lock

sleep mutex

no

spin for a while, then sleep

lock manager

yes

sleep lock

Highest

witness

yes

partially ordered sleep locks

Spin locks are built from the hardware test-and-set instruction. A memory location is reserved for the lock with a value of zero showing that the lock is free and a value of one showing that the lock is held. The test-and-set instruction tries to acquire the lock. The lock value is tested and the lock unconditionally set to one. If the tested value is zero, then the lock was successfully acquired and the thread may proceed. If the value is one, then some other thread held the lock so the thread must loop doing the test-and-set until the thread holding the lock (and running on a different processor) stores a zero into the lock to show that it is done with it. Spin locks are used for locks that are held only briefly—for example, to protect a list while adding or removing an entry from it.

It is wasteful of CPU cycles to use spin locks for resources that will be held for long periods of time. For example, a spin lock would be inappropriate for a disk buffer that would need to be locked throughout the time that a disk I/O was being done. Here a sleep lock should be used. When a thread trying to acquire a sleep-type lock finds that the lock is held, it is put to sleep so that other threads can run until the lock becomes available.

The time to acquire a lock can be variable—for example, a lock needed to search and remove an item from a list. The list usually will be short, for which a spin lock would be appropriate, but will occasionally grow long. Here a hybrid lock is used; the lock spins for a while, but if unsuccessful after a specified number of attempts, it reverts to a sleep-type lock. Most architectures require 100 to 200 instructions to put a thread to sleep and then awaken it again. A spin lock that can be acquired in less than this time is going to be more efficient than a sleep lock. The hybrid lock is usually set to try for about half this time before reverting to a sleep lock.

Spin locks are never appropriate on a uniprocessor, since the only way that a resource held by another thread will ever be released will be when that thread gets to run. Thus, spin locks are always converted to sleep locks when running on a uniprocessor.

The highest-level locking prevents threads from deadlocking when locking multiple resources. Suppose that two threads, A and B, require exclusive access to two resources, Rl and R2, to do some operation as shown in Figure 4.4. If thread A acquires Rl and thread B acquires R2, then a deadlock occurs when thread A tries to acquire R2 and thread B tries to acquire Rl. To avoid deadlock, FreeBSD 5.2 maintains a partial ordering on all the locks. The two partial-ordering rules are as follows:

  1. A thread may acquire only one lock in each class.

  2. A thread may acquire only a lock in a higher-numbered class than the highest-numbered class for which it already holds a lock.

Figure 4.4. Partial ordering of resources.

In Figure 4.4 thread A holds R1 and can request R2 as R1 and R2 are in different classes and R2 is in a higher-numbered class than R1. However, thread B must release R2 before requesting R1, since R2 is in a higher class than R1. Thus, thread A will be able to acquire R2 when it is released by thread B. After thread A completes and releases R1 and R2, thread B will be able to acquire both of those locks and run to completion without deadlock.

Historically, the class members and ordering were poorly documented and unenforced. Violations were discovered when threads would deadlock and a careful analysis done to figure out what ordering had been violated. With an increasing number of developers and a growing kernel, the ad hoc method of maintaining the partial ordering of locks became untenable. A witness module was added to the kernel to derive and enforce the partial ordering of the locks. The witness module keeps track of the locks acquired and released by each thread. It also keeps track of the order in which locks are acquired relative to each other. Each time a lock is acquired, the witness module uses these two lists to verify that a lock is not being acquired in the wrong order. If a lock order violation is detected, then a message is output to the console detailing the locks involved and the locations in question. The witness module also verifies that no spin locks are held when requesting a sleep lock or voluntarily going to sleep.

The witness module can be configured to either panic or drop into the kernel debugger when an order violation occurs or some other witness check fails. When running the debugger, the witness module can output the list of locks held by the current thread to the console along with the filename and line number at which each lock was last acquired. It can also dump the current order list to the console. The code first displays the lock order tree for all the sleep locks. Then it displays the lock order tree for all the spin locks. Finally, it displays a list of locks that have not yet been acquired.

Mutex Synchronization

Mutexes are the primary method of thread synchronization. The major design considerations for mutexes are

  • Acquiring and releasing uncontested mutexes should be as cheap as possible.

  • Mutexes must have the information and storage space to support priority propagation.

  • A thread must be able to recursively acquire a mutex if the mutex is initialized to support recursion.

There are currently two flavors of mutexes: those that sleep and those that do not. By default, mutexes will sleep when they are already held. As a machine-dependent optimization they may spin for some amount of time before sleeping. Most kernel code uses the default lock type; the default lock type allows the thread to be disconnected from the CPU if it cannot get the lock. The machine-dependent implementation may treat the lock as a short-term spin lock under some circumstances. However, it is always safe to use these forms of locks in an interrupt thread without fear of deadlock against an interrupted thread on the same CPU. If a thread interrupted the thread that held a mutex and then tried to acquire the mutex, it would be put to sleep until the thread holding the mutex ran to completion and released the mutex.

Mutexes that do not sleep are called spin mutexes. A spin mutex will not relinquish the CPU when it cannot immediately get the requested lock, but it will loop, waiting for the mutex to be released by another CPU. This spinning can result in deadlock if a thread interrupted the thread that held a mutex and then tried to acquire the mutex. To protect an interrupt service routine from blocking against itself all interrupts are blocked on the local processor while holding a spin lock. Thus, the interrupt can run only on another processor during the period that the mutex is held.

Spin locks are specialized locks that are intended to be held for short periods of time; their primary purpose is to protect portions of the code that implement default (i.e., sleep) locks. These mutexes should only be used to protect data shared with any devices that require nonpreemptive interrupts and low-level scheduling code. On most architectures both acquiring and releasing an uncontested spin mutex is more expensive than the same operation on a nonspin mutex. It is permissible to hold multiple spin mutexes. Here, it is a required that they be released in the opposite order to that in which they were acquired. It is not permissible to go to sleep while holding a spin mutex.

The mtx_init() function must be used to initialize a mutex before it can be passed to mtx_lock(). The mtx_init() function specifies a type that the witness code uses to classify a mutex when doing checks of lock ordering. It is not permissible to pass the same mutex to mtx_init() multiple times without intervening calls to mtx_destroy().

The mtx_lock() function acquires a mutual exclusion lock for the currently running kernel thread. If another kernel thread is holding the mutex, the caller will sleep until the mutex is available. The mtx_lock_spin() function is similar to the mtx_lock() function except that it will spin until the mutex becomes available. Interrupts are disabled on the CPU holding the mutex during the spin and remain disabled following the acquisition of the lock.

It is possible for the same thread to recursively acquire a mutex with no ill effects if the MTX_RECURSE bit was passed to mtx_init() during the initialization of the mutex. The witness module verifies that a thread does not recurse on a non-recursive lock. A recursive lock is useful if a resource may be locked at two or more levels in the kernel. By allowing a recursive lock, a lower layer need not check if the resource has already been locked by a higher layer; it can simply lock and release the resource as needed.

The mtx_trylock() function tries to acquire a mutual exclusion lock for the currently running kernel thread. If the mutex cannot be immediately acquired, mtx_trylock() will return 0; otherwise the mutex will be acquired and a nonzero value will be returned.

The mtx_unlock() function releases a mutual exclusion lock; if a higher-priority thread is waiting for the mutex, the releasing thread will be put to sleep to allow the higher-priority thread to acquire the mutex and run. A mutex that allows recursive locking maintains a reference count showing the number of times that it has been locked. Each successful lock request must have a corresponding unlock request. The mutex is not released until the final unlock has been done, causing the reference count to drop to zero.

The mtx_unlock_spin( function releases a spin-type mutual exclusion lock; interrupt state from before the lock was acquired is restored.

The mtx_destroy() function destroys a mutex so the data associated with it may be freed or otherwise overwritten. Any mutex that is destroyed must previously have been initialized with mtx_init(). It is permissible to have a single reference to a mutex when it is destroyed. It is not permissible to hold the mutex recursively or have another thread blocked on the mutex when it is destroyed. If these rules are violated, the kernel will panic.

The giant lock that protects subsystems in FreeBSD 5.2 that have not yet been converted to operate on a multiprocessor must be acquired before acquiring other mutexes. Put another way: It is impossible to acquire giant nonrecursively while holding another mutex. It is possible to acquire other mutexes while holding giant, and it is possible to acquire giant recursively while holding other mutexes.

Sleeping while holding a mutex (except for giant) is almost never safe and should be avoided. There are numerous assertions that will fail if this is attempted.

Lock-Manager Locks

Interprocess synchronization to a resource typically is implemented by associating it with a lock structure. The kernel has a lock manager that manipulates a lock. The operations provided by the lock manager are

  • Request shared: Get one of many possible shared locks. If a thread holding an exclusive lock requests a shared lock, the exclusive lock will be downgraded to a shared lock.

  • Request exclusive: Stop further shared locks when they are cleared, grant a pending upgrade (see following) if it exists, and then grant an exclusive lock. Only one exclusive lock may exist at a time, except that a thread holding an exclusive lock may get additional exclusive locks if it explicitly sets the canrecurse flag in the lock request or if the canrecurse flag was set when the lock was initialized.

  • Request upgrade: The thread must hold a shared lock that it wants to have upgraded to an exclusive lock. Other threads may get exclusive access to the resource between the time that the upgrade is requested and the time that it is granted.

  • Request exclusive upgrade: The thread must hold a shared lock that it wants to have upgraded to an exclusive lock. If the request succeeds, no other threads will have gotten exclusive access to the resource between the time that the upgrade is requested and the time that it is granted. However, if another thread has already requested an upgrade, the request will fail.

  • Request downgrade: The thread must hold an exclusive lock that it wants to have downgraded to a shared lock. If the thread holds multiple (recursive) exclusive locks, they will all be downgraded to shared locks.

  • Request release: Release one instance of a lock.

  • Request drain: Wait for all activity on the lock to end, and then mark it decommissioned. This feature is used before freeing a lock that is part of a piece of memory that is about to be released.

Locks must be initialized before their first use by calling the lockinit() function. Parameters to the lockinit() function include the following:

  • A top-half kernel priority at which the thread should run once it has acquired the lock

  • Flags such as canrecurse that allows the thread currently holding an exclusive lock to get another exclusive lock rather than panicking with a "locking against myself" failure

  • A string that describes the resource that the lock protects referred to as the wait channel message

  • An optional maximum time to wait for the lock to become available

Other Synchronization

In addition to the general lock manager locks and mutexes, there are three other types of locking available for use in the kernel. These other types of locks are less heavily used, but they are included here for completeness.

Shared/exclusive locks are used to protect data that are read far more often than they are written. Mutexes are inherently more efficient than shared/exclusive locks. However, shared/exclusive locks are faster than lock-manager locks because they lack many of the lock-manager lock features such as the ability to be upgraded or downgraded.

Condition variables are used with mutexes to wait for conditions to occur. Threads wait on condition variables by calling cv_wait(), cv_wait_sig() (wait unless interrupted by a signal), cv_timedwait() (wait for a maximum time), or cv_timedwait_sig() (wait unless interrupted by a signal or for a maximum time). Threads unblock waiters by calling cv_signal() to unblock one waiter, or cv_broadcast() to unblock all waiters. The cv_waitq_remove() function removes a waiting thread from a condition variable wait queue if it is on one.

A thread must hold a mutex before calling cv_wait(), cv_wait_sig(), cv_timedwait(), or cv_timedwait_sig(). When a thread waits on a condition, the mutex is atomically released before the thread is blocked, and then atomically reacquired before the function call returns. All waiters must use the same mutex with a condition variable. A thread must hold the mutex while calling cv_signal() or cv_broadcast().

Counting semaphores provide a mechanism for synchronizing access to a pool of resources. Unlike mutexes, semaphores do not have the concept of an owner, so they can also be useful in situations where one thread needs to acquire a resource and another thread needs to release it. Each semaphore has an integer value associated with it. Posting (incrementing) always succeeds, but waiting (decrementing) can only successfully complete if the resulting value of the semaphore is greater than or equal to zero. Semaphores are not used where mutexes and condition variables will suffice. Semaphores are a more complex synchronization mechanism than mutexes and condition variables, and are not as efficient.


4.4. Thread Scheduling

Traditionally, the FreeBSD scheduler had an ill-defined set of hooks spread through the kernel. In FreeBSD 5.0, these hooks were regularized and a well-defined API was created so that different schedulers could be developed. Since FreeBSD 5.0, the kernel has had two schedulers available:

  • The historic 4.4BSD scheduler found in the file /sys/kern/sched_4bsd.c. This scheduler is described at the beginning of this section.

  • The new ULE scheduler first introduced in FreeBSD 5.0 and found in the file /sys/kern/sched_ule.c [Roberson, 2003]. The name is not an acronym. If the underscore in its file name is removed, the rationale for its name becomes apparent. This scheduler is described at the end of this section.

Because a busy system makes thousands of scheduling decisions per second, the speed with which scheduling decisions are made is critical to the performance of the system as a whole. Other UNIX systems have added a dynamic scheduler switch that must be traversed for every scheduling decision. To avoid this overhead, FreeBSD requires that the scheduler be selected at the time the kernel is built. Thus, all calls into the scheduling code are resolved at compile time rather than going through the overhead of an indirect function call for every scheduling decision. By default, kernels up through FreeBSD 5.1 use the 4.4BSD scheduler. Beginning with FreeBSD 5.2, the ULE scheduler is used by default.

The 4.4BSD Scheduler

All threads that are runnable are assigned a scheduling priority that determines in which run queue they are placed. In selecting a new thread to run, the system scans the run queues from highest to lowest priority and chooses the first thread on the first nonempty queue. If multiple threads reside on a queue, the system runs them round robin; that is, it runs them in the order that they are found on the queue, with equal amounts of time allowed. If a thread blocks, it is not put back onto any run queue. If a thread uses up the time quantum (or time slice) allowed it, it is placed at the end of the queue from which it came, and the thread at the front of the queue is selected to run.

The shorter the time quantum, the better the interactive response. However, longer time quanta provide higher system throughput because the system will have less overhead from doing context switches and processor caches will be flushed less often. The time quantum used by FreeBSD is 0.1 second. This value was empirically found to be the longest quantum that could be used without loss of the desired response for interactive jobs such as editors. Perhaps surprisingly, the time quantum has remained unchanged over the past 20 years. Although the time quantum was originally selected on centralized timesharing systems with many users, it is still correct for decentralized workstations today. While workstation users expect a response time faster than that anticipated by the timesharing users of 20 years ago, the shorter run queues on the typical workstation makes a shorter quantum unnecessary.

Time-Share Thread Scheduling

The FreeBSD time-share-scheduling algorithm is based on multilevel feedback queues. The system adjusts the priority of a thread dynamically to reflect resource requirements (e.g., being blocked awaiting an event) and the amount of resources consumed by the thread (e.g., CPU time). Threads are moved between run queues based on changes in their scheduling priority (hence the word feedback in the name multilevel feedback queue). When a thread other than the currently running thread attains a higher priority (by having that priority either assigned or given when it is awakened), the system switches to that thread immediately if the current thread is in user mode. Otherwise, the system switches to the higher-priority thread as soon as the current thread exits the kernel. The system tailors this short-term scheduling algorithm to favor interactive jobs by raising the scheduling priority of threads that are blocked waiting for I/O for 1 or more seconds and by lowering the priority of threads that accumulate significant amounts of CPU time.

Short-term thread scheduling is broken up into two parts. The next section describes when and how a thread's scheduling priority is altered; the section after that describes the management of the run queues and the interaction between thread scheduling and context switching.

Calculations of Thread Priority

A thread's scheduling priority is determined directly by two values associated with the thread structure: kg_estcpu and kg_nice. The value of kg_estcpu provides an estimate of the recent CPU utilization of the thread. The value of kg_nice is a user-settable weighting factor that ranges numerically between -20 and 20. The normal value for kg_nice is 0. Negative values increase a thread's priority, whereas positive values decrease its priority.

A thread's user-mode scheduling priority is calculated after every four clock ticks (typically 40 milliseconds) that it has been found running by this equation:

Equation 4.1

Values less than PRI_MIN_TIMESHARE (160) are set to PRI_MIN_TIMESHARE (see Table 4.2); values greater than PRI_MAX_TIMESHARE (223) are set to PRI_MAX_TIMESHARE. This calculation causes the priority to decrease linearly based on recent CPU utilization. The user-controllable kg_nice parameter acts as a limited weighting factor. Negative values retard the effect of heavy CPU utilization by offsetting the additive term containing kg_estcpu. Otherwise, if we ignore the second term, kg_nice simply shifts the priority by a constant factor.

The CPU utilization, kg_estcpu, is incremented each time that the system clock ticks and the thread is found to be executing. In addition, kg_estcpu is adjusted once per second via a digital decay filter. The decay causes about 90 percent of the CPU usage accumulated in a 1-second interval to be forgotten over a period of time that is dependent on the system load average. To be exact, kg_estcpu is adjusted according to

Equation 4.2

where the load is a sampled average of the sum of the lengths of the run queue and of the short-term sleep queue over the previous 1-minute interval of system operation.

To understand the effect of the decay filter, we can consider the case where a single compute-bound thread monopolizes the CPU. The thread's CPU utilization will accumulate clock ticks at a rate dependent on the clock frequency. The load average will be effectively 1, resulting in a decay of

If we assume that the thread accumulates Ti clock ticks over time interval i and that kg_nice is zero, then the CPU utilization for each time interval will count into the current value of kg estcpu according to

Thus, after five decay calculations, only 13 percent of T0 remains present in the current CPU utilization value for the thread. Since the decay filter is applied once per second, we can also say that about 90 percent of the CPU utilization is forgotten after 5 seconds.

Threads that are runnable have their priority adjusted periodically as just described. However, the system ignores threads blocked awaiting an event: These threads cannot accumulate CPU usage, so an estimate of their filtered CPU usage can be calculated in one step. This optimization can significantly reduce a system's scheduling overhead when many blocked threads are present. The system recomputes a thread's priority when that thread is awakened and has been sleeping for longer than 1 second. The system maintains a value, kg_slptime, that is an estimate of the time a thread has spent blocked waiting for an event. The value of kg_slptime is set to 0 when a thread calls sleep() and is incremented once per second while the thread remains in a SLEEPING or STOPPED state. When the thread is awakened, the system computes the value of kg_estcpu according to

Equation 4.3

and then recalculates the scheduling priority using Eq. 4.1. This analysis ignores the influence of kg_nice; also, the load used is the current load average rather than the load average at the time that the thread blocked.

Thread-Priority Routines

The priority calculations used in the short-term scheduling algorithm are spread out in several areas of the system. Two routines, schedcpu() and roundrobin(), run periodically. Schedcpu() recomputes thread priorities once per second, using Eq. 4.2, and updates the value of kg_slptime for threads blocked by a call to sleep(). The roundrobin() routine runs 10 times per second and causes the system to reschedule the threads in the highest-priority (nonempty) queue in a round-robin fashion, which allows each thread a 100-millisecond time quantum.

The CPU usage estimates are updated in the system clock-processing module, hardclock(), which executes 100 times per second. Each time that a thread accumulates four ticks in its CPU usage estimate, kg_estcpu, the system recalculates the priority of the thread. This recalculation uses Eq. 4.1 and is done by the resetpriority() routine. The decision to recalculate after four ticks is related to the management of the run queues described in the next section. In addition to issuing the call from hardclock(), each time setrunnable() places a thread on a run queue, it also calls resetpriority() to recompute the thread's scheduling priority. This call from wakeup() to setrunnable() operates on a thread other than the currently running thread. So setrunnable() invokes updatepri() to recalculate the CPU usage estimate according to Eq. 4.3 before calling resetpriority(). The relationship of these functions is shown in Figure 4.5.

Figure 4.5. Procedural interface to priority calculation.

Thread Run Queues and Context Switching

The kernel has a single set of run queues to manage all the thread scheduling classes shown in Table 4.2. The scheduling-priority calculations described in the previous section are used to order the set of timesharing threads into the priority ranges between 160 and 223. The real-time threads and the idle threads priorities are set by the applications themselves but are constrained by the kernel to be within the ranges 128 to 159 and 224 to 255, respectively. The number of queues used to hold the collection of all runnable threads in the system affects the cost of managing the queues. If only a single (ordered) queue is maintained, then selecting the next runnable thread becomes simple but other operations become expensive. Using 256 different queues can significantly increase the cost of identifying the next thread to run. The system uses 64 run queues, selecting a run queue for a thread by dividing the thread's priority by 4. To save time, the threads on each queue are not further sorted by their priorities.

The run queues contain all the runnable threads in main memory except the currently running thread. Figure 4.6 (on page 104) shows how each queue is organized as a doubly linked list of thread structures. The head of each run queue is kept in an array. Associated with this array is a bit vector, rq_status, that is used in identifying the nonempty run queues. Two routines, runq_add() and runq_remove (), are used to place a thread at the tail of a run queue, and to take a thread off the head of a run queue. The heart of the scheduling algorithm is the runq_choose() routine. The runq_choose() routine is responsible for selecting a new thread to run; it operates as follows:

  1. Ensure that our caller acquired the sched_lock.

  2. Locate a nonempty run queue by finding the location of the first nonzero bit in the rq_status bit vector. If rq_status is zero, there are no threads to run, so select the idle loop thread.

  3. Given a nonempty run queue, remove the first thread on the queue.

  4. If this run queue is now empty as a result of removing the thread, reset the appropriate bit in rq_status.

  5. Return the selected thread.

Figure 4.6. Queueing structure for runnable threads.

[View full size image]

The context-switch code is broken into two parts. The machine-independent code resides in mi_switch(); the machine-dependent part resides in cpu_switch(). On most architectures, cpu_switch() is coded in assembly language for efficiency.

Given the mi_switch() routine and the thread-priority calculations, the only missing piece in the scheduling facility is how the system forces an involuntary context switch. Remember that voluntary context switches occur when a thread calls the sleep() routine. Sleep() can be invoked by only a runnable thread, so sleep() needs only to place the thread on a sleep queue and to invoke mi_switch() to schedule the next thread to run. Often an interrupt thread will not want to sleep() itself but will be delivering data that will cause the kernel to want to run a different thread than the one that was running before the interrupt. Thus, the kernel needs a mechanism to request that an involuntary context switch be done at the conclusion of the interrupt.

This mechanism is handled by setting the currently running thread's TDF_NEEDRESCHED flag and then posting an asynchronous system trap (AST). An AST is a trap that is delivered to a thread the next time that that thread is preparing to return from an interrupt, a trap, or a system call. Some architectures support ASTs directly in hardware; other systems emulate ASTs by checking an AST flag at the end of every system call, trap, and interrupt. When the hardware AST trap occurs or the AST flag is set, the mi_switch() routine is called, instead of the current thread resuming execution. Rescheduling requests are made by the swi_sched(), resetpriority(), setrunnable(), wake up(), roundrobin(), and schedcpu() routines.

With the advent of multiprocessor support FreeBSD can preempt threads executing in kernel mode. However, such preemption is generally not done, so the worst-case real-time response to events is defined by the longest path through the top half of the kernel. Since the system guarantees no upper bounds on the duration of a system call, FreeBSD is decidedly not a real-time system. Attempts to retrofit BSD with real-time thread scheduling have addressed this problem in different ways [Ferrin & Langridge, 1980; Sanderson et al., 1986].

The ULE Scheduler

The ULE scheduler was developed as part of the overhaul of FreeBSD to support SMP. A new scheduler was undertaken for several reasons:

  • To address the need for processor affinity in SMP systems

  • To provide better support for symmetric multithreading (SMT)—processors with multiple, on chip, CPU cores

  • To improve the performance of the scheduling algorithm so that it is no longer dependent on the number of threads in the system

The goal of a multiprocessor system is to apply the power of multiple CPUs to a problem, or set of problems, to achieve a result in less time than it would run on a single-processor system. If a system has the same number of runnable threads as it does CPUs, then achieving this goal is easy. Each runnable thread gets a CPU to itself and runs to completion. Typically, there are many runnable threads competing for a few processors. One job of the scheduler is to ensure that the CPUs are always busy and are not wasting their cycles. When a thread completes its work, or is blocked waiting for resources, it is removed from the processor on which it was running. While a thread is running on a processor, it brings its working set—the instructions it is executing and the data on which it is operating—into the memory cache of the CPU. Migrating a thread has a cost. When a thread is moved from one processor to another, its in-cache working set is lost and must be removed from the processor on which it was running and then loaded into the new CPU to which it has been migrated. The performance of an SMP system with a naive scheduler that does not take this cost into account can fall beneath that of a single-processor system. The term processor affinity describes a scheduler that only migrates threads when necessary to give an idle processor something to do.

Many microprocessors now provide support for symmetric multithreading where the processor is built out of multiple CPU cores, each of which can execute a thread. The CPU cores in an SMT processor share all the processor's resources, such as memory caches and access to main memory, so they are more tightly synchronized than the processors in an SMP system. From a thread's perspective, it does not know that there are other threads running on the same processor because the processor is handling them independently. The one piece of code in the system that needs to be aware of the multiple cores is the scheduling algorithm. The SMT case is a slightly different version of the processor affinity problem presented by an SMP system. Each CPU core can be seen as a processor with its own set of threads. In an SMP system composed of CPUs that support SMT, the scheduler treats each core on a processor as a less powerful resource but one to which it is cheaper to migrate threads.

The original FreeBSD scheduler maintains a global list of threads that it traverses once per second to recalculate their priorities. The use of a single list for all threads means that the performance of the scheduler is dependent on the number of tasks in the system, and as the number of tasks grow, more CPU time must be spent in the scheduler maintaining the list. A design goal of the ULE scheduler was to avoid the need to consider all the runnable threads in the system to make a scheduling decision.

The ULE scheduler creates a set of three queues for each CPU in the system. Having per-processor queues makes it possible to implement processor affinity in an SMP system.

One queue is the idle queue, where all idle threads are stored. The other two queues are designated current and next. Threads are picked to run, in priority order, from the current queue until it is empty, at which point the current and next queues are swapped and scheduling is started again. Threads in the idle queue are run only when the other two queues are empty. Real-time and interrupt threads are always inserted into the current queue so that they will have the least possible scheduling latency. Interactive threads are also inserted into the current queue to keep the interactive response of the system acceptable. A thread is considered to be interactive if the ratio of its voluntary sleep time versus its run time is below a certain threshold. The interactivity threshold is defined in the ULE code and is not configurable. ULE uses two equations to compute the interactivity score of a thread. For threads whose sleep time exceeds their run time Eq 4.4 is used:

Equation 4.4

When a thread's run time exceeds its sleep time, Eq. 4.5 is used instead.

Equation 4.5

The scaling factor is the maximum interactivity score divided by two. Threads that score below the interactivity threshold are considered to be interactive; all others are noninteractive. The sched_interact_update() routine is called at several points in a threads existence—for example when the thread is awakened by a wakeup() call—to update the thread's run time and sleep time. The sleep and runtime values are only allowed to grow to a certain limit. When the sum of the run time and sleep time pass the limit, they are reduced to bring them back into range. An interactive thread whose sleep history was not remembered at all would not remain interactive, resulting in a poor user experience. Remembering an interactive thread's sleep time for too long would allow the thread to more than its fair share of the CPU. The amount of history that is kept and the interactivity threshold are the two values that most strongly influence a user's interactive experience on the system.

Noninteractive threads are put into the next queue and are scheduled to run when the queues are switched. Switching the queues guarantees that a thread gets to run at least once every two queue switches regardless of priority, which ensures fair sharing of the processor.

There are two mechanisms used to migrate threads among multiple processors. When a CPU has no work to do in any of its queues, it marks a bit in a bit-mask shared by all processors that says that it is idle. Whenever an active CPU is about to add work to its own run queue, it first checks to see if it has excess work and if another processor in the system is idle. If an idle processor is found, then the thread is migrated to the idle processor using an interprocessor interrupt (IPI). Making a migration decision by inspecting a shared bitmask is much faster than scanning the run queues of all the other processors. Seeking out idle processors when adding a new task works well because it spreads the load when it is presented to the system.

The second form of migration, called push migration, is done by the system on a periodic basis and more aggressively offloads work to other processors in the system. Twice per second the sched_balance() routine picks the most-loaded and least-loaded processors in the system and equalizes their run queues. The balancing is done only between two processors because it was thought that two processor systems would be the most common and to prevent the balancing algorithm from being too complex and adversely affecting the performance of the scheduler. Push migration ensures fairness among the runnable threads. For example, with three runnable threads on a two-processor system, it would be unfair for one thread to get a processor to itself while the other two had to share the second processor. By pushing a thread from the processor with two threads to the processor with one thread, no single thread would get to run alone indefinitely.

Handling the SMT case is a derivative form of load balancing among full-fledged CPUs and is handled by processor groups. Each CPU core in an SMT processor is given its own kseq structure, and these structures are grouped under a kseq group structure. An example of a single processor with two cores is shown in Figure 4.7 (on page 108). In an SMP system with multiple SMT capable processors there would be one processor group per CPU. When the scheduler is deciding to which processor or core to migrate a thread, it will try to pick a core on the same processor before picking one on another processor because that is the lowest-cost migration path.

Figure 4.7. Processor with two cores.


4.5. Process Creation

In FreeBSD, new processes are created with the fork family of system calls. The fork system call creates a complete copy of the parent process. The rfork system call creates a new process entry that shares a selected set of resources from its parent rather than making copies of everything. The vfork system call differs from fork in how the virtual-memory resources are treated; vfork also ensures that the parent will not run until the child does either an exec or exit system call. The vfork system call is described in Section 5.6.

The process created by a fork is termed a child process of the original parent process. From a user's point of view, the child process is an exact duplicate of the parent process except for two values: the child PID and the parent PID. A call to fork returns the child PID to the parent and zero to the child process. Thus, a program can identify whether it is the parent or child process after a fork by checking this return value.

A fork involves three main steps:

1. Allocating and initializing a new process structure for the child process

2. Duplicating the context of the parent (including the thread structure and virtual-memory resources) for the child process

3. Scheduling the child process to run

The second step is intimately related to the operation of the memory-management facilities described in Chapter 5. Consequently, only those actions related to process management will be described here.

The kernel begins by allocating memory for the new process and thread entries (see Figure 4.1). These thread and process entries are initialized in three steps: One part is copied from the parent's corresponding structure, another part is zeroed, and the rest is explicitly initialized. The zeroed fields include recent CPU utilization, wait channel, swap and sleep time, timers, tracing, and pending-signal information. The copied portions include all the privileges and limitations inherited from the parent, including the following:

  • The process group and session

  • The signal state (ignored, caught, and blocked signal masks)

  • The kg_nice scheduling parameter

  • A reference to the parent's credential

  • A reference to the parent's set of open files

  • A reference to the parent's limits

The explicitly set information includes

  • Entry onto the list of all processes

  • Entry onto the child list of the parent and the back pointer to the parent

  • Entry onto the parent's process-group list

  • Entry onto the hash structure that allows the process to be looked up by its PID

  • A pointer to the process's statistics structure, allocated in its user structure

  • A pointer to the process's signal-actions structure, allocated in its user structure

  • A new PID for the process

The new PID must be unique among all processes. Early versions of BSD verified the uniqueness of a PID by performing a linear search of the process table. This search became infeasible on large systems with many processes. FreeBSD maintains a range of unallocated PIDs between lastpid and pidchecked. It allocates a new PID by incrementing and then using the value of lastpid. When the newly selected PID reaches pidchecked, the system calculates a new range of unused PIDs by making a single scan of all existing processes (not just the active ones are scanned—zombie and swapped processes also are checked).

The final step is to copy the parent's address space. To duplicate a process's image, the kernel invokes the memory-management facilities through a call to vm_forkproc(). The vm_forkproc() routine is passed a pointer to the initialized process structure for the child process and is expected to allocate all the resources that the child will need to execute. The call to vm_forkproc() returns through a different execution path directly into user mode in the child process and via the normal execution path in the parent process.

Once the child process is fully built, its thread is made known to the scheduler by being placed on the run queue. The alternate return path will set the return value of fork system call in the child to 0. The normal execution return path in the parent sets the return value of the fork system call to be the new PID.


4.6. Process Termination

Processes terminate either voluntarily through an exit system call or involuntarily as the result of a signal. In either case, process termination causes a status code to be returned to the parent of the terminating process (if the parent still exists). This termination status is returned through the wait4 system call. The wait4 call permits an application to request the status of both stopped and terminated processes. The wait4 request can wait for any direct child of the parent, or it can wait selectively for a single child process or for only its children in a particular process group. Wait4 can also request statistics describing the resource utilization of a terminated child process. Finally, the wait4 interface allows a process to request status codes without blocking.

Within the kernel, a process terminates by calling the exit() routine. The exit() routine first kills off any other threads associated with the process. The termination of other threads is done as follows:

  • Any thread entering the kernel from user space will thread_exit() when it traps into the kernel.

  • Any thread already in the kernel and attempting to sleep will return immediately with EINTR or EAGAIN, which will force them to back out to user space, freeing resources as they go. When the thread attempts to return to user space, it will instead hit thread_exit().

The exit() routine then cleans up the process's kernel-mode execution state by doing the following:

  • Canceling any pending timers

  • Releasing virtual-memory resources

  • Closing open descriptors

  • Handling stopped or traced child processes

With the kernel-mode state reset, the process is then removed from the list of active processes—the allproc list—and is placed on the list of zombie processes pointed to by zombproc. The process state is changed to show that no thread is currently running. The exit() routine then does the following:

  • Records the termination status in the p_xstat field of the process structure

  • Bundles up a copy of the process's accumulated resource usage (for accounting purposes) and hangs this structure from the p_ru field of the process structure

  • Notifies the deceased process's parent

Finally, after the parent has been notified, the cpu_exit() routine frees any machine-dependent process resources and arranges for a final context switch from the process.

The wait4 call works by searching a process's descendant processes for processes that have terminated. If a process in ZOMBIE state is found that matches the wait criterion, the system will copy the termination status from the deceased process. The process entry then is taken off the zombie list and is freed. Note that resources used by children of a process are accumulated only as a result of a wait4 system call. When users are trying to analyze the behavior of a long-running program, they would find it useful to be able to obtain this resource usage information before the termination of a process. Although the information is available inside the kernel and within the context of that program, there is no interface to request it outside that context until process termination.


4.7. Signals

UNIX defines a set of signals for software and hardware conditions that may arise during the normal execution of a program; these signals are listed in Table 4.4 (on page 112). Signals may be delivered to a process through application-specified signal handlers or may result in default actions, such as process termination, carried out by the system. FreeBSD signals are designed to be software equivalents of hardware interrupts or traps.

Table 4.4. Signals defined in FreeBSD.

Name

Default action

Description

SIGHUP

terminate process

terminal line hangup

SIGINT

terminate process

interrupt program

SIGQUIT

create core image

quit program

SIGILL

create core image

illegal instruction

SIGTRAP

create core image

trace trap

SIGABRT

create core image

abort

SIGEMT

create core image

emulate instruction executed

SIGFPE

create core image

floating-point exception

SIGKILL

terminate process

kill program

SIGBUS

create core image

bus error

SIGSEGV

create core image

segmentation violation

SIGSYS

create core image

bad argument to system call

SIGPIPE

terminate process

write on a pipe with no one to read it

SIGALRM

terminate process

real-time timer expired

SIGTERM

terminate process

software termination signal

SIGURG

discard signal

urgent condition on I/O channel

SIGSTOP

stop process

stop signal not from terminal

SIGTSTP

stop process

stop signal from terminal

SIGCONT

discard signal

a stopped process is being continued

SIGCHLD

discard signal

notification to parent on child stop or exit

SIGTTIN

stop process

read on terminal by background process

SIGTTOU

stop process

write to terminal by background process

SIGIO

discard signal

I/O possible on a descriptor

SIGXCPU

terminate process

CPU time limit exceeded

SIGXFSZ

terminate process

file-size limit exceeded

SIGVTALRM

terminate process

virtual timer expired

SIGPROF

terminate process

profiling timer expired

SIGWINCH

discard signal

window size changed

SIGINFO

discard signal

information request

SIGUSR1

terminate process

user-defined signal 1

SIGUSR2

terminate process

user-defined signal 2

Each signal has an associated action that defines how it should be handled when it is delivered to a process. If a process contains more than one thread, each thread may specify whether it wishes to take action for each signal. Typically, one thread elects to handle all the process-related signals such as interrupt, stop, and continue. All the other threads in the process request that the process-related signals be masked out. Thread-specific signals such as segmentation fault, floating point exception, and illegal instruction are handled by the thread that caused them. Thus, all threads typically elect to receive these signals. The precise disposition of signals to threads is given in the later subsection on posting a signal. First, we describe the possible actions that can be requested.

The disposition of signals is specified on a per-process basis. If a process has not specified an action for a signal, it is given a default action (see Table 4.4) that may be any one of the following:

  • Ignoring the signal

  • Terminating all the threads in the process

  • Terminating all the threads in the process after generating a core file that contains the process's execution state at the time the signal was delivered

  • Stopping all the threads in the process

  • Resuming the execution of all the threads in the process

An application program can use the sigaction system call to specify an action for a signal, including these choices:

  • Taking the default action

  • Ignoring the signal

  • Catching the signal with a handler

A signal handler is a user-mode routine that the system will invoke when the signal is received by the process. The handler is said to catch the signal. The two signals SIGSTOP and SIGKILL cannot be masked, ignored, or caught; this restriction ensures that a software mechanism exists for stopping and killing runaway processes. It is not possible for a process to decide which signals would cause the creation of a core file by default, but it is possible for a process to prevent the creation of such a file by ignoring, blocking, or catching the signal.

Signals are posted to a process by the system when it detects a hardware event, such as an illegal instruction, or a software event, such as a stop request from the terminal. A signal may also be posted by another process through the kill system call. A sending process may post signals to only those receiving processes that have the same effective user identifier (unless the sender is the superuser). A single exception to this rule is the continue signal, SIGCONT, which always can be sent to any descendant of the sending process. The reason for this exception is to allow users to restart a setuid program that they have stopped from their keyboard.

Like hardware interrupts, each thread in a process can mask the delivery of signals. The execution state of each thread contains a set of signals currently masked from delivery. If a signal posted to a thread is being masked, the signal is recorded in the thread's set of pending signals, but no action is taken until the signal is unmasked. The sigprocmask system call modifies a set of masked signals for a thread. It can add to the set of masked signals, delete from the set of masked signals, or replace the set of masked signals. Although the delivery of the SIGCONT signal to the signal handler of a process may be masked, the action of resuming that stopped process is not masked.

Two other signal-related system calls are sigsuspered and sigaltstack. The sigsuspend call permits a thread to relinquish the processor until that thread receives a signal. This facility is similar to the system's sleep() routine. The sigaltstack call allows a process to specify a run-time stack to use in signal delivery. By default, the system will deliver signals to a process on the latter's normal run-time stack. In some applications, however, this default is unacceptable. For example, if an application has many threads that have carved up the normal run-time stack into many small pieces, it is far more memory efficient to create one large signal stack on which all the threads handle their signals than it is to reserve space for signals on each thread's stack.

The final signal-related facility is the sigreturn system call. Sigreturn is the equivalent of a user-level load-processor-context operation. A pointer to a (machine-dependent) context block that describes the user-level execution state of a thread is passed to the kernel. The sigreturn system call restores state and resumes execution after a normal return from a user's signal handler.

History of Signals

Signals were originally designed to model exceptional events, such as an attempt by a user to kill a runaway program. They were not intended to be used as a general interprocess-communication mechanism, and thus no attempt was made to make them reliable. In earlier systems, whenever a signal was caught, its action was reset to the default action. The introduction of job control brought much more frequent use of signals and made more visible a problem that faster processors also exacerbated: If two signals were sent rapidly, the second could cause the process to die, even though a signal handler had been set up to catch the first signal. Thus, reliability became desirable, so the developers designed a new framework that contained the old capabilities as a subset while accommodating new mechanisms.

The signal facilities found in FreeBSD are designed around a virtual-machine model, in which system calls are considered to be the parallel of machine's hardware instruction set. Signals are the software equivalent of traps or interrupts, and signal-handling routines do the equivalent function of interrupt or trap service routines. Just as machines provide a mechanism for blocking hardware interrupts so that consistent access to data structures can be ensured, the signal facilities allow software signals to be masked. Finally, because complex run-time stack environments may be required, signals, like interrupts, may be handled on an alternate application-provided run-time stack. These machine models are summarized in Table 4.5.

Table 4.5. Comparison of hardware-machine operations and the corresponding software virtual-machine operations.

Hardware Machine

Software Virtual Machine

instruction set

set of system calls

restartable instructions

restartable system calls

interrupts/traps

signals

interrupt/trap handlers

signal handlers

blocking interrupts

masking signals

interrupt stack

signal stack

Posting of a Signal

The implementation of signals is broken up into two parts: posting a signal to a process and recognizing the signal and delivering it to the target thread. Signals may be posted by any process or by code that executes at interrupt level. Signal delivery normally takes place within the context of the receiving thread. But when a signal forces a process to be stopped, the action can be carried out on all the threads associated with that process when the signal is posted.

A signal is posted to a single process with the psignal() routine or to a group of processes with the gsignal() routine. The gsignal() routine invokes psignal() for each process in the specified process group. The actions associated with posting a signal are straightforward, but the details are messy. In theory, posting a signal to a process simply causes the appropriate signal to be added to the set of pending signals for the appropriate thread within the process, and the selected thread is then set to run (or is awakened if it was sleeping at an interruptible priority level).

The disposition of signals is set on a per-process basis. So the kernel first checks to see if the signal should be ignored in which case it is discarded. If the process has specified the default action, then the default action is taken. If the process has specified a signal handler that should be run, then the kernel must select the appropriate thread within the process that should handle the signal. When a signal is raised because of the action of the currently running thread (for example, a segment fault), the kernel will only try to deliver it to that thread. If the thread is masking the signal, then the signal will be held pending until it is unmasked. When a process-related signal is sent (for example, an interrupt), then the kernel searches all the threads associated with the process, searching for one that does not have the signal masked. The signal is delivered to the first thread that is found with the signal unmasked. If all threads associated with the process are masking the signal, then the signal is left in the list of signals pending for the process for later delivery.

The cursig() routine calculates the next signal, if any, that should be delivered to a thread. It determines the next signal by inspecting the process's signal list, p_siglist, to see if it has any signals that should be propagated to the thread's signal list, td_siglist. It then inspects the td_siglist field to check for any signals that should be delivered to the thread. Each time that a thread returns from a call to sleep() (with the PCATCH flag set) or prepares to exit the system after processing a system call or trap, it checks to see whether a signal is pending delivery. If a signal is pending and must be delivered in the thread's context, it is removed from the pending set, and the thread invokes the postsig() routine to take the appropriate action.

The work of psignal() is a patchwork of special cases required by the process-debugging and job-control facilities and by intrinsic properties associated with signals. The steps involved in posting a signal are as follows:

  1. Determine the action that the receiving process will take when the signal is delivered. This information is kept in the p_sigignore and p_sigcatch fields of the process's process structure. If a process is not ignoring or catching a signal, the default action is presumed to apply. If a process is being traced by its parent—that is, by a debugger—the parent process is always permitted to intercede before the signal is delivered. If the process is ignoring the signal, psignal()'s work is done and the routine can return.

  2. Given an action, p_signal() selects the appropriate thread and adds the signal to the thread's set of pending signals, td_siglist, and then does any implicit actions specific to that signal. For example, if the signal is a continue signal, SIGCONT, any pending signals that would normally cause the process to stop, such as SIGTTOU, are removed.

  3. Next, psignal() checks whether the signal is being masked. If the thread is currently masking delivery of the signal, psignal()'s work is complete and it may return.

  4. If, however, the signal is not being masked, psignal() must either do the action directly or arrange for the thread to execute so that the thread will take the action associated with the signal. Before setting the thread to a runnable state, psignal() must take different courses of action depending on the thread state as follows:

    SLEEPING

    The thread is blocked awaiting an event. If the thread is sleeping noninterruptibly, then nothing further can be done. Otherwise, the kernel can apply the action — either directly or indirectly — by waking up the thread. There are two actions that can be applied directly. For signals that cause a process to stop, all the threads in the process are placed in the STOPPED state, and the parent process is notified of the state change by a SIGCHLD signal being posted to it. For signals that are ignored by default, the signal is removed from the signal list and the work is complete. Otherwise, the action associated with the signal must be done in the context of the receiving thread, and the thread is placed onto the run queue with a call to setrunnable().

    STOPPED

    The process is stopped by a signal or because it is being debugged. If the process is being debugged, then there is nothing to do until the controlling process permits it to run again. If the process is stopped by a signal and the posted signal would cause the process to stop again, then there is nothing to do, and the posted signal is discarded.

    Otherwise, the signal is either a continue signal or a signal that would normally cause the process to terminate (unless the signal is caught). If the signal is SIGCONT, then all the threads in the process that were previously running are set running again. Any threads in the process that were blocked waiting on an event arc returned to the SLEEPING state. If the signal is SIGKILL, then all the threads in the process are set running again no matter what, so that they can terminate the next time that they are scheduled to run. Otherwise, the signal causes the threads in the process to be made mnnable, but the threads are not placed on the run queue because they must wait for a continue signal.

    RUNNABLE, NEW, ZOMBIE

    If a thread scheduled to receive a signal is not the currently executing thread, its TDF_NEEDRESCHED flag is set, so that the signal will be noticed by the receiving thread as soon as possible.

Delivering a Signal

Most actions associated with delivering a signal to a thread are carried out within the context of that thread. A thread checks its td_siglist field for pending signals at least once each time that it enters the system, by calling cursig().

If cursig() determines that there are any unmasked signals in the thread's signal list, it calls issignal() to find the first unmasked signal in the list. If delivering the signal causes a signal handler to be invoked or a core dump to be made, the caller is notified that a signal is pending, and the delivery is done by a call to postsig (). That is,

if (sig = cursig(curthread))
    postsig(sig);

Otherwise, the action associated with the signal is done within issignal() (these actions mimic the actions carried out by psignal()).

The postsig() routine has two cases to handle:

  1. Producing a core dump

  2. Invoking a signal handler

The former task is done by the coredump() routine and is always followed by a call to exit() to force process termination. To invoke a signal handler, postsig() first calculates a set of masked signals and installs that set in td_sigmask. This set normally includes the signal being delivered, so that the signal handler will not be invoked recursively by the same signal. Any signals specified in the sigaction system call at the time the handler was installed also will be included. The postsig() routine then calls the sendsig() routine to arrange for the signal handler to execute immediately after the thread returns to user mode. Finally, the signal in td_siglist is cleared and postsig() returns, presumably to be followed by a return to user mode.

The implementation of the sendsig() routine is machine dependent. Figure 4.8 shows the flow of control associated with signal delivery. If an alternate stack has been requested, the user's stack pointer is switched to point at that stack. An argument list and the thread's current user-mode execution context are stored by the kernel on the (possibly new) stack. The state of the thread is manipulated so that, on return to user mode, a call will be made immediately to a body of code termed the signal-trampoline code. This code invokes the signal handler (between steps 2 and 3 in Figure 4.8) with the appropriate argument list, and, if the handler returns, makes a sigreturn system call to reset the thread's signal state to the state that existed before the signal.

Figure 4.8. Delivery of a signal to a process. Step 1: The kernel places a signal context on the user's stack. Step 2: The kernel places a signal-handler frame on the user's stack and arranges to start running the user process in the sigtramp() code. When the sigtramp() routine starts running, it calls the user's signal handler. Step 3: The user's signal handler returns to the sigtramp() routine, which pops the signal-handler context from the user's stack. Step 4: The sigtramp() routine finishes by calling the sigretum system call, which restores the previous user context from the signal context, pops the signal context from the stack, and resumes the user's process at the point at which it was running before the signal occurred.

[View full size image]


4.8. Process Groups and Sessions

A process group is a collection of related processes, such as a shell pipeline, all of which have been assigned the same process-group identifier. The process-group identifier is the same as the PID of the process group's initial member; thus process-group identifiers share the name space of process identifiers. When a new process group is created, the kernel allocates a process-group structure to be associated with it. This process-group structure is entered into a process-group hash table so that it can be found quickly.

A process is always a member of a single process group. When it is created, each process is placed into the process group of its parent process. Programs such as shells create new process groups, usually placing related child processes into a group. A process can change its own process group or that of a child process by creating a new process group or by moving a process into an existing process group using the setpgid system call. For example, when a shell wants to set up a new pipeline, it wants to put the processes in the pipeline into a process group different from its own so that the pipeline can be controlled independently of the shell. The shell starts by creating the first process in the pipeline, which initially has the same process-group identifier as the shell. Before executing the target program, the first process does a setpgid to set its process-group identifier to the same value as its PID. This system call creates a new process group, with the child process as the process-group leader of the process group. As the shell starts each additional process for the pipeline, each child process uses setpgid to join the existing process group.

In our example of a shell creating a new pipeline, there is a race condition. As the additional processes in the pipeline are spawned by the shell, each is placed in the process group created by the first process in the pipeline. These conventions are enforced by the setpgid system call. It restricts the set of process-group identifiers to which a process may be set to either a value equal its own PID or a value of another process-group identifier in its session. Unfortunately, if a pipeline process other than the process-group leader is created before the process-group leader has completed its setpgid call, the setpgid call to join the process group will fail. As the setpgid call permits parents to set the process group of their children (within some limits imposed by security concerns), the shell can avoid this race by making the setpgid call to change the child's process group both in the newly created child and in the parent shell. This algorithm guarantees that, no matter which process runs first, the process group will exist with the correct process-group leader. The shell can also avoid the race by using the vfork variant of the fork system call that forces the parent process to wait until the child process either has done an exec system call or has exited. In addition, if the initial members of the process group exit before all the pipeline members have joined the group—for example, if the process-group leader exits before the second process joins the group, the setpgid call could fail. The shell can avoid this race by ensuring that all child processes are placed into the process group without calling the wait system call, usually by blocking the SIGCHLD signal so that the shell will not be notified yet if a child exits. As long as a process-group member exists, even as a zombie process, additional processes can join the process group.

There are additional restrictions on the setpgid system call. A process may join process groups only within its current session (discussed in the next section), and it cannot have done an exec system call. The latter restriction is intended to avoid unexpected behavior if a process is moved into a different process group after it has begun execution. Therefore, when a shell calls setpgid in both parent and child processes after a fork, the call made by the parent will fail if the child has already made an exec call. However, the child will already have joined the process group successfully, and the failure is innocuous.

Sessions

Just as a set of related processes are collected into a process group, a set of process groups are collected into a session. A session is a set of one or more process groups and may be associated with a terminal device. The main uses for sessions are to collect together a user's login shell and the jobs that it spawns and to create an isolated environment for a daemon process and its children. Any process that is not already a process-group leader may create a session using the setsid system call, becoming the session leader and the only member of the session. Creating a session also creates a new process group, where the process-group ID is the PID of the process creating the session, and the process is the process-group leader. By definition, all members of a process group are members of the same session.

A session may have an associated controlling terminal that is used by default for communicating with the user. Only the session leader may allocate a controlling terminal for the session, becoming a controlling process when it does so. A device can be the controlling terminal for only one session at a time. The terminal I/O system (described in Chapter 10) synchronizes access to a terminal by permitting only a single process group to be the foreground process group for a controlling terminal at any time. Some terminal operations are restricted to members of the session. A session can have at most one controlling terminal. When a session is created, the session leader is dissociated from its controlling terminal if it had one.

A login session is created by a program that prepares a terminal for a user to log into the system. That process normally executes a shell for the user, and thus the shell is created as the controlling process. An example of a typical login session is shown in Figure 4.9.

Figure 4.9. A session and its processes. In this example, process 3 is the initial member of the session—the session leader—and is referred to as the controlling process if it has a controlling terminal. It is contained in its own process group, 3. Process 3 has spawned two jobs: One is a pipeline composed of processes 4 and 5, grouped together in process group 4, and the other one is process 8, which is in its own process group, 8. No process-group leader can create a new session; thus, processes 3, 4, or 8 could not start its own session, but process 5 would be allowed to do so.

The data structures used to support sessions and process groups in FreeBSD are shown in Figure 4.10. This figure parallels the process layout shown in Figure 4.9. The pg_members field of a process-group structure heads the list of member processes; these processes are linked together through the p_pglist list entry in the process structure. In addition, each process has a reference to its process-group structure in the p_pgrp field of the process structure. Each process-group structure has a pointer to its enclosing session. The session structure tracks per-login information, including the process that created and controls the session, the controlling terminal for the session, and the login name associated with the session. Two processes wanting to determine whether they are in the same session can traverse their p-pgrp pointers to find their process-group structures and then compare the pg_session pointers to see whether the latter are the same.

Figure 4.10. Process-group organization.

[View full size image]

Job Control

Job control is a facility first provided by the C shell [Joy, 1994] and today provided by most shells. It permits a user to control the operation of groups of processes termed jobs. The most important facilities provided by job control are the abilities to suspend and restart jobs and to do the multiplexing of access to the user's terminal. Only one job at a time is given control of the terminal and is able to read from and write to the terminal. This facility provides some of the advantages of window systems, although job control is sufficiently different that it is often used in combination with window systems. Job control is implemented on top of the process group, session, and signal facilities.

Each job is a process group. Outside the kernel, a shell manipulates a job by sending signals to the job's process group with the killpg system call, which delivers a signal to all the processes in a process group. Within the system, the two main users of process groups are the terminal handler (Chapter 10) and the interprocess-communication facilities (Chapter 11). Both facilities record process-group identifiers in private data structures and use them in delivering signals. The terminal handler, in addition, uses process groups to multiplex access to the controlling terminal.

For example, special characters typed at the keyboard of the terminal (e.g., control-C or control-\) result in a signal being sent to all processes in one job in the session; that job is in the foreground, whereas all other jobs in the session are in the background. A shell may change the foreground job by using the tcsetpgrp() function, implemented by the TIOCSPGRP ioctl on the controlling terminal. Background jobs will be sent the SIGTTIN signal if they attempt to read from the terminal, normally stopping the job. The SIGTTOU signal is sent to background jobs that attempt an ioctl system call that would alter the state of the terminal, and, if the TOSTOP option is set for the terminal, if they attempt to write to the terminal.

The foreground process group for a session is stored in the t_pgrp field of the session's controlling terminal tty structure (see Chapter 10). All other process groups within the session are in the background. In Figure 4.10, the session leader has set the foreground process group for its controlling terminal to be its own process group. Thus, its two jobs are in the background, and the terminal input and output will be controlled by the session-leader shell. Job control is limited to processes contained within the same session and to the terminal associated with the session. Only the members of the session are permitted to reassign the controlling terminal among the process groups within the session.

If a controlling process exits, the system revokes further access to the controlling terminal and sends a SIGHUP signal to the foreground process group. If a process such as a job-control shell exits, each process group that it created will become an orphaned process group: a process group in which no member has a parent that is a member of the same session but of a different process group. Such a parent would normally be a job-control shell capable of resuming stopped child processes. The pg_jobc field in Figure 4.10 counts the number of processes within the process group that have the controlling process as a parent. When that count goes to zero, the process group is orphaned. If no action were taken by the system, any orphaned process groups that were stopped at the time that they became orphaned would be unlikely ever to resume. Historically, the system dealt harshly with such stopped processes: They were killed. In POSIX and FreeBSD, an orphaned process group is sent a hangup and a continue signal if any of its members are stopped when it becomes orphaned by the exit of a parent process. If processes choose to catch or ignore the hangup signal, they can continue running after becoming orphaned. The system keeps a count of processes in each process group that have a parent process in another process group of the same session. When a process exits, this count is adjusted for the process groups of all child processes. If the count reaches zero, the process group has become orphaned. Note that a process can be a member of an orphaned process group even if its original parent process is still alive. For example, if a shell starts a job as a single process A, that process then forks to create process B, and the parent shell exits; then process B is a member of an orphaned process group but is not an orphaned process.

To avoid stopping members of orphaned process groups if they try to read or write to their controlling terminal, the kernel does not send them SIGTTIN and SIGTTOU signals, and prevents them from stopping in response to those signals. Instead, attempts to read or write to the terminal produce an error.


4.9. Jails

The FreeBSD access control mechanism is designed for an environment with two types of users: those with and those without administrative privilege. It is often desirable to delegate some but not all administrative functions to untrusted or less trusted parties and simultaneously impose systemwide mandatory policies on process interaction and sharing. Historically, attempting to create such an environment has been both difficult and costly. The primary mechanism for partial delegation of administrative authority is to write a set-user-identifier program that carefully controls which of the administrative privileges may be used. These set-user-identifier programs are complex to write, difficult to maintain, limited in their flexibility, and are prone to bugs that allow undesired administrative privilege to be gained.

Many operating systems attempt to address these limitations by providing fine-grained access controls for system resources [P1003.1e, 1998]. These efforts vary in degrees of success, but almost all suffer from at least three serious limitations:

  1. Increasing the granularity of security controls increases the complexity of the administration process, in turn increasing both the opportunity for incorrect configuration, as well as the demand on administrator time and resources. Often the increased complexity results in significant frustration for the administrator, which may result in two disastrous types of policy: running with security features disabled and running with the default configuration on the assumption that it will be secure.

  2. Usefully segregating capabilities and assigning them to running code and users is difficult. Many privileged operations in FreeBSD seem independent but are interrelated. The handing out of one privilege may be transitive to many others. For example, the ability to mount filesystems allows new set-user-identifier programs to be made available that in turn may yield other unintended security capabilities.

  3. Introducing new security features often involves introducing new security management interfaces. When fine-grained capabilities are introduced to replace the set-user-identifier mechanism in FreeBSD, applications that previously did an appropriateness check to see if they were running with superuser privilege before executing must now be changed to know that they need not run with superuser privilege. For applications running with privilege and executing other programs, there is now a new set of privileges that must be voluntarily given up before executing another program. These changes can introduce significant incompatibility for existing applications and make life more difficult for application developers who may not be aware of differing security semantics on different systems.

This abstract risk becomes more clear when applied to a practical, real-world example: many Web service providers use FreeBSD to host customer Web sites. These providers must protect the integrity and confidentiality of their own files and services from their customers. They must also protect the files and services of one customer from (accidental or intentional) access by any other customer. A provider would like to supply substantial autonomy to customers, allowing them to install and maintain their own software and to manage their own services such as Web servers and other content-related daemon programs.

This problem space points strongly in the direction of a partitioning solution. By putting each customer in a separate partition, customers are isolated from accidental or intentional modification of data or disclosure of process information from customers in other partitions. Delegation of management functions within the system must be possible without violating the integrity and privacy protection between partitions.

FreeBSD-style access control makes it notoriously difficult to compartmentalize functionality. While mechanisms such as chroot provide a modest level of compartmentalization, this mechanism has serious shortcomings, both in the scope of its functionality and effectiveness at what it provides. The chroot system call was first added to provide an alternate build environment for the system. It was later adapted to isolate anonymous ftp access to the system.

The original intent of chroot was not to ensure security. Even when used to provide security for anonymous ftp, the set of operations allowed by ftp was carefully controlled to prevent those that allowed escape from the chroot''ed environment.

Three classes of escape from the confines of a chroot-created filesystem were identified over the years:

  1. Recursive chroot escapes

  2. Escapes using..

  3. Escapes using fchdir

All these escapes exploited the lack of enforcement of the new root directory.

Two changes to chroot were made to detect and thwart these escapes. To prevent the first two escapes, the directory of the first level of chroot experienced by a process is recorded. Any attempts to traverse backward across this directory are refused. The third escape using fchdir is prevented by having the chroot system call fail if the process has any file descriptors open referencing directories.

Even with stronger semantics, the chroot system call is insufficient to provide complete partitioning. Its compartmentalization does not extend to the process or networking spaces. Therefore, both observation of and interference with processes outside their compartment is possible. To provide a secure virtual machine environment, FreeBSD added a new "jail" facility built on top of chroot. Processes in a jail are provided full access to the files that they may manipulate, processes they may influence, and network services they may use. They are denied access to and visibility of files, processes, and network services outside their jail [Kamp & Watson, 2000].

Unlike other fine-grained security solutions, a jail does not substantially increase the policy management requirements for the system administrator. Each jail is a virtual FreeBSD environment that permits local policy to be independently managed. The environment within a jail has the same properties as the host system. Thus, a jail environment is familiar to the administrator and compatible with applications [Hope, 2002].

Jail Semantics

Two important goals of the jail implementation are to:

  1. Retain the semantics of the existing discretionary access-control mechanisms

  2. Allow each jail to have its own superuser administrator whose activities are limited to the processes, files, and network associated with its jail

The first goal retains compatibility with most applications. The second goal permits the administrator of a FreeBSD machine to partition the host into separate jails and provide access to the superuser account in each of these jails without losing control of the host environment.

A process in a partition is referred to as being "in jail." When FreeBSD first boots, no processes will be jailed. Jails are created when a privileged process calls the jail system call with arguments of the filesystem into which it should chroot and the IP address and hostname to be associated with the jail. The process that creates the jail will be the first and only process placed in the jail. Any future descendants of the jailed process will be in its jail. A process may never leave a jail that it created or in which it was created. A process may be in only one jail. The only way for a new process to enter the jail is by inheriting access to the jail from another process already in that jail.

Each jail is bound to a single IP address. Processes within the jail may not make use of any other IP address for outgoing or incoming connections. A jail has the ability to restrict the set of network services that it chooses to offer at its address. An application request to bind all IP addresses are redirected to the individual address associated of the jail in which the requesting process is running.

A jail takes advantage of the existing chroot behavior to limit access to the filesystem name space for jailed processes. When a jail is created, it is bound to a particular filesystem root. Processes are unable to manipulate files that they cannot address. Thus, the integrity and confidentiality of files outside the jail filesystem root are protected.

Processes within the jail will find that they are unable to interact or even verify the existence of processes outside the jail. Processes within the jail are prevented from delivering signals to processes outside the jail, connecting to processes outside the jail with debuggers, or even seeing processes outside the jail with the usual system monitoring mechanisms. Jails do not prevent, nor are they intended to prevent, the use of covert channels or communications mechanisms via accepted interfaces. For example, two processes in different jails may communicate via sockets over the network. Jails do not attempt to provide scheduling services based on the partition.

Jailed processes are subject to the normal restrictions present for any processes including resource limits and limits placed by the network code, including firewall rules. By specifying firewall rules for the IP address bound to a jail, it is possible to place connectivity and bandwidth limitations on that jail, restricting the services that it may consume or offer.

The jail environment is a subset of the host environment. The jail filesystem appears as part of the host filesystem and may be directly modified by processes in the host environment. Processes within the jail appear in the process listing of the host and may be signalled or debugged.

Processes running without superuser privileges will notice few differences between a jailed environment or an unjailed environment. Standard system services such remote login and mail servers behave normally as do most third-party applications, including the popular Apache Web server.

Processes running with superuser privileges will find that many restrictions apply to the privileged calls they may make. Most of the limitations are designed to restrict activities that would affect resources outside the jail. These restrictions include prohibitions against the following:

  • Modifying the running kernel by direct access or loading kernel modules

  • Mounting and unmounting filesystems

  • Creating device nodes

  • Modifying kernel run-time parameters such as most sysctl settings

  • Changing security-level flags

  • Modifying any of the network configuration, interfaces, addresses, and routing-table entries

  • Accessing raw, divert, or routing sockets. These restrictions prevent access to facilities that allow spoofing of IP numbers or the generation of disruptive traffic.

  • Accessing network resources not associated with the jail. Specifically, an attempt to bind a reserved port number on all available addresses will result in binding only the address associated with the jail.

  • Administrative actions that would affect the host system such as rebooting

Other privileged activities are permitted as long as they are limited to the scope of the jail:

  • Signalling any process within the jail is permitted.

  • Deleting or changing the ownership and mode of any file within the jail is permitted, as long as the file flags permit the requested change.

  • The superuser may read a file owned by any UID, as long as it is accessible through the jail filesystem name space.

  • Binding reserved TCP and UDP port numbers on the jail's IP address is permitted.

These restrictions on superuser access limit the scope of processes running with superuser privileges, enabling most applications to run unhindered but preventing calls that might allow an application to reach beyond the jail and influence other processes or systemwide configuration.

Jail Implementation

The implementation of the jail system call is straightforward. A prison data structure is allocated and populated with the arguments provided. The prison structure is linked to the process structure of the calling process. The prison structure's reference count is set to one, and the chroot system call is called to set the jail's root. The prison structure may not be modified once it is created.

Hooks in the code implementing process creation and destruction maintain the reference count on the prison structure and free it when the last reference is released. Any new processes created by a process in a jail will inherit a reference to the prison structure, which puts the new process in the same jail.

Some changes were needed to restrict process visibility and interaction. The kernel interfaces that report running processes were modified to report only the processes in the same jail as the process requesting the process information. Determining whether one process may send a signal to another is based on UID and GID values of the sending and receiving processes. With jails, the kernel adds the requirement that if the sending process is jailed, then the receiving process must be in the same jail.

Several changes were added to the networking implementation:

  • Restricting TCP and UDP access to just one IP number was done almost entirely in the code that manages protocol control blocks (see Section 13.1). When a jailed process binds to a socket, the IP number provided by the process will not be used; instead, the preconfigured IP number of the jail is used.

  • The loop-back interface, which has the magic IP number 127.0.0.1, is used by processes to contact servers on the local machine. When a process running in a jail connects to the 127.0.0.1 address, the kernel must intercept and redirect the connection request to the IP address associated with the jail.

  • The interfaces through which the network configuration and connection state may be queried were modified to report only information relevant to the configured IP number of a jailed process.

Device drivers for shared devices such as the pseudo-terminal driver (see Section 10.1) needed to be changed to enforce that a particular virtual terminal cannot be accessed from more than one jail at the same time.

The simplest but most tedious change was to audit the entire kernel for places that allowed the superuser extra privilege. Only about 35 of the 300 checks in FreeBSD 5.2 were opened to jailed processes running with superuser privileges. Since the default is that jailed superusers do not receive privilege, new code or drivers are automatically jail-aware: They will refuse jailed superusers privilege.

Jail Limitations

As it stands, the jail code provides a strict subset of system resources to the jail environment, based on access to processes, files, network resources, and privileged services. Making the jail environment appear to be a fully functional FreeBSD system allows maximum application support and the ability to offer a wide range of services within a jail environment. However, there are limitations in the current implementation. Removing these limitations will enhance the ability to offer services in a jail environment. Three areas that deserve greater attention are the set of available network resources, management of scheduling resources, and support for orderly jail shutdown.

Currently, only a single IP version 4 address may be allocated to each jail, and all communication from the jail is limited to that IP address. It would be desirable to support multiple addresses or possibly different address families for each jail. Access to raw sockets is currently prohibited, as the current implementation of raw sockets allows access to raw IP packets associated with all interfaces. Limiting the scope of the raw socket would allow its safe use within a jail, thus allowing the use of ping and other network debugging and evaluation tools.

Another area of great interest to the current users of the jail code is the ability to limit the effect of one jail on the CPU resources available for other jails. Specifically, they require that the system have ways to allocate scheduling resources among the groups of processes in each of the jails. Work in the area of lottery scheduling might be leveraged to allow some degree of partitioning between jail environments [Petrou & Milford, 1997].

Management of jail environments is currently somewhat ad hoc. Creating and starting jails is a well-documented procedure, but jail shutdown requires the identification and killing of all the processes running within the jail. One approach to cleaning up this interface would be to assign a unique jail-identifier at jail creation time. A new jailkill system call would permit the direction of signals to specific jail-identifiers, allowing for the effective termination of all processes in the jail. FreeBSD makes use of an init process to bring the system up during the boot process and to assist in shutdown (see Section 14.6). A similarly operating process, jailinit, running in each jail would present a central location for delivering management requests to its jail from the host environment or from within the jail. The jailinit process would coordinate the clean shutdown of the jail before resorting to terminating processes, in the same style as the host environment shutting down before killing all processes and halting the kernel.


4.10. Process Debugging

FreeBSD provides a simplistic facility for controlling and debugging the execution of a process. This facility, accessed through the ptrace system call, permits a parent process to control a child process's execution by manipulating user- and kernel-mode execution state. In particular, with ptrace, a parent process can do the following operations on a child process:

  • Attach to an existing process to begin debugging it

  • Read and write address space and registers

  • Intercept signals posted to the process

  • Single step and continue the execution of the process

  • Terminate the execution of the process

The ptrace call is used almost exclusively by program debuggers, such as gdb.

When a process is being traced, any signals posted to that process cause it to enter the STOPPED state. The parent process is notified with a SIGCHLD signal and may interrogate the status of the child with the wait4 system call. On most machines, trace traps, generated when a process is single stepped, and breakpoint faults, caused by a process executing a breakpoint instruction, are translated by FreeBSD into SIGTRAP signals. Because signals posted to a traced process cause it to stop and result in the parent being notified, a program's execution can be controlled easily.

To start a program that is to be debugged, the debugger first creates a child process with a fork system call. After the fork, the child process uses a ptrace call that causes the process to be flagged as traced by setting the P_TRACED bit in the p_flag field of the process structure. The child process then sets the trace trap bit in the process's processor status word and calls execve to load the image of the program that is to be debugged. Setting this bit ensures that the first instruction executed by the child process after the new image is loaded will result in a hardware trace trap, which is translated by the system into a SIGTRAP signal. Because the parent process is notified about all signals to the child, it can intercept the signal and gain control over the program before it executes a single instruction.

Alternatively, the debugger may take over an existing process by attaching to it. A successful attach request causes the process to enter the STOPPED state and to have its P_TRACED bit set in the p_flag field of its process structure. The debugger can then begin operating on the process in the same way as it would with a process that it had explicitly started.

An alternative to the ptrace system call is the /proc filesystem. The functionality provided by the /proc filesystem is the same as that provided by ptrace; it differs only in its interface. The /proc filesystem implements a view of the system process table inside the filesystem and is so named because it is normally mounted on /proc. It provides a two-level view of process space. At the highest level, processes themselves are named, according to their process IDs. There is also a special node called curproc that always refers to the process making the lookup request.

Each node is a directory that contains the following entries:

ctl | A write-only file that supports a variety of control operations. Control commands are written as strings to the ctl file. The control commands are:
 | attach | Stops the target process and arranges for the sending process to become the debug control process.
 | detach | Continue execution of the target process and remove it from control by the debug process (that need not be the sending process).
 | Run | Continue running the target process until a signal is delivered, a breakpoint is hit, or the target process exits.
 | Step | Single step the target process, with no signal delivery.
 | wait | Wait for the target process to come to a steady state ready for debugging. The target process must be in this state before any of the other commands are allowed.
 | The string can also be the name of a signal, lowercase and without the SIG prefix, in which case that signal is delivered to the process.
dbregs | Set the debug registers as defined by the machine architecture.
etype | The type of the executable referenced by the file entry.
file | A reference to the vnode from which the process text was read. This entry can be used to gain access to the symbol table for the process or to start another copy of the process.
fpregs | The floating point registers as defined by the machine architecture. It is only implemented on machines that have distinct general purpose and floating point register sets.
map | A map of the process's virtual memory.
mem | The complete virtual memory image of the process. Only those addresses that exist in the process can be accessed. Reads and writes to this file modify the process. Writes to the text segment remain private to the process. Because the address space of another process can be accessed with read and write system calls, a debugger can access a process being debugged with much greater efficiency than it can with the ptrace system call. The pages of interest in the process being debugged are mapped into the kernel address space. The data requested by the debugger can then becopied directly from the kernel to the debugger's address space.
regs | Allows read and write access to the register set of the process.
rlimit | A read-only file containing the process current and maximum limits.
status | The process status. This file is read-only and returns a single line containing multiple space-separated fields that include the command name,the process id, the parent process id, the process group id, the session id, the controlling terminal (if any), a list of the process flags, the process start time, user and system times, the wait channel message, and the process credentials.

Each node is owned by the process's user and belongs to that user's primary group, except for the mem node, which belongs to the kmem group.

In a normal debugging environment, where the target does a fork followed by an exec by the debugger, the debugger should fork and the child should stop itself (with a self-inflicted SIGSTOP, for example). The parent should issue a wait and then an attach command via the appropriate ctl file. The child process will receive a SIGTRAP immediately after the call to exec.


Exercises

4.1 For each state listed in Table 4.1, list the system queues on which a process in that state might be found.

4.2 Why is the performance of the context-switching mechanism critical to the performance of a highly multiprogrammed system?

4.3 What effect would increasing the time quantum have on the system's interactive response and total throughput?

4.4 What effect would reducing the number of run queues from 64 to 32 have on the scheduling overhead and on system performance?

4.5 Give three reasons for the system to select a new process to run.

4.6 Describe the three types of scheduling policies provided by FreeBSD.

4.7 What type of jobs does the time-share scheduling policy favor? Propose an algorithm for identifying these favored jobs.

4.8 When and how does thread scheduling interact with memory-management facilities?

4.9 After a process has exited, it may enter the state of being a ZOMBIE before disappearing from the system entirely. What is the purpose of the ZOMBIE state? What event causes a process to exit from ZOMBIE?

4.10 Suppose that the data structures shown in Figure 4.2 do not exist. Instead, assume that each process entry has only its own PID and the PID of its parent. Compare the costs in space and time to support each of the following operations:

  1. Creation of a new process

  2. Lookup of the process's parent

  3. Lookup of all of a process's siblings

  4. Lookup of all of a process's descendants

  5. Destruction of a process

4.11 What are the differences between a mutex and a lock-manager lock?

4.12 Give an example of where a mutex lock should be used. Give an example of where a lock-manager lock should be used.

4.13 A process blocked without setting the PCATCH flag may never be awakened by a signal. Describe two problems a noninterruptible sleep may cause if a disk becomes unavailable while the system is running.

4.14 Describe the limitations a jail puts on the nlesystem name space, network access, and processes running in the jail.

*4.15 In FreeBSD, the signal SIGTSTP is delivered to a process when a user types a "suspend character." Why would a process want to catch this signal before it is stopped?

*4.16 Before the FreeBSD signal mechanism was added, signal handlers to catch the SIGTSTP signal were written as

catchstop()
{
    prepare to stop;
    signal (SIGTSTP, SIG_DFL);
    kill (getpid(), SIGTSTP);
    signal (SIGTSTP, catchstop) ;
}

This code causes an infinite loop in FreeBSD. Why does it do so? How should the code be rewritten?

*4.17 The process-priority calculations and accounting statistics are all based on sampled data. Describe hardware support that would permit more accurate statistics and priority calculations.

*4.18 Why are signals a poor interprocess-communication facility?

**4.19 A kernel-stack-invalid trap occurs when an invalid value for the kernel-mode stack pointer is detected by the hardware. How might the system gracefully terminate a process that receives such a trap while executing on its kernel-run-time stack?

**4.20 Describe alternatives to the test-and-set instruction that would allow you to build a synchronization mechanism for a multiprocessor FreeBSD system.

**4.21 A lightweight process is a thread of execution that operates within the context of a normal FreeBSD process. Multiple lightweight processes may exist in a single FreeBSD process and share memory, but each is able to do blocking operations, such as system calls. Describe how lightweight processes might be implemented entirely in user mode.


References

Aral et al., 1989. Z. Aral, J. Bloom, T. Doeppner, I. Gertner, A. Langerman, & G. Schaffer, "Variable Weight Processes with Flexible Shared Resources," USENIX Association Conference Proceedings, pp. 405-12, January 1989.

Ferrin & Langridge, 1980. T. E. Ferrin & R. Langridge, "Interactive Computer Graphics with the UNIX Time-Sharing System," Computer Graphics, vol. 13, pp. 320-331, 1980.

Hope, 2002. P. Hope, "Using Jails in FreeBSD for Fun and Profit," ;login: The USENIX Association Newsletter, vol. 27, no. 3, pp. 48-55, available from http://www.usenix.org/publications/login/2002-06/pdfs/hope.pdf, USENIX Association, Berkeley, CA, June 2002.

Joy, 1994. W. N. Joy, "An Introduction to the C Shell," in 4.4BSD User's Supplementary Documents, pp. 4:1_16, O'Reilly & Associates, Inc., Sebastopol, CA, 1994.

Kamp & Watson, 2000. P. Kamp & R. Watson, "Jails: Confining the Omnipotent Root," Proceedings of the Second International System Administration and Networking Conference (SANE), available from http://docs.freebsd.org/44doc/papers/jail/, May 2000.

PlOOS.le, 1998. P1003.1e, Unpublished Draft Standard for Information Technology— Portable Operating System Interface (POSIX)—Part 1: System Application Program Interface—Amendment: Protection, Audit and Control Interfaces [C Language] IEEE Standard 1003.le Draft 17 Editor Casey Schaufler, Institute of Electrical and Electronic Engineers, Piscataway, NJ, 1998.

Petrou & Milford, 1997. D. Petrou & J. Milford, Proportional-Share Scheduling: Implementation and Evaluation in a Widely-Deployed Operating System, available from http://www.cs.cmu.edu/~dpetrou/papers/freebsd_lottery_writeup98.ps and http://www.cs.cmu.edu/~dpetrou/code/freebsd_lottery_code.tar.gz, 1997.

Ritchie, 1988. D. M. Ritchie, "Multi-Processor UNIX," private communication, April 25, 1988.

Roberson, 2003. J. Roberson, "ULE: A Modern Scheduler For FreeBSD," Proceedings of BSDCon 2003, September 2003.

Sanderson et al., 1986. T. Sanderson, S. Ho, N. Heijden, E. Jabs, & J. L. Green, "Near-Realtime Data Transmission During the ICE-Comet Giacobini-Zinner Encounter," ESA Bulletin, vol. 45, no. 21, 1986.

Schimmel, 1994. C. Schimmel, UNIX Systems for Modern Architectures, Symmetric Multiprocessing, and Caching for Kernel Programmers, Addison-Wesley, Reading, MA, 1994.


Chapter 5. Memory Management


5.1. Terminology

A central component of any operating system is the memory-management system. As the name implies, memory-management facilities are responsible for the management of memory resources available on a machine. These resources are typically layered in a hierarchical fashion, with memory-access times inversely related to their proximity to the CPU (see Figure 5.1 on page 136). The primary memory system is main memory; the next level of storage is secondary storage or backing storage. Main-memory systems usually are constructed from random-access memories, whereas secondary stores are placed on moving-head disk drives. In certain workstation environments, the common two-level hierarchy is becoming a three-level hierarchy, with the addition of file-server machines or network-attached storage connected to a workstation via a local-area network [Gingell, Moran, & Shannon, 1987].

Figure 5.1. Hierarchical layering of memory.

[View full size image]

In a multiprogrammed environment, it is critical for the operating system to share available memory resources effectively among the processes. The operation of any memory-management policy is directly related to the memory required for a process to execute. That is, if a process must reside entirely in main memory for it to execute, then a memory-management system must be oriented toward allocating large units of memory. On the other hand, if a process can execute when it is only partially resident in main memory, then memory-management policies are likely to be substantially different. Memory-management facilities usually try to optimize the number of runnable processes that are resident in main memory. This goal must be considered with the goals of the process scheduler (Chapter 4) so that conflicts that can adversely affect overall system performance are avoided.

Although the availability of secondary storage permits more processes to exist than can be resident in main memory, it also requires additional algorithms that can be complicated. Space management typically requires algorithms and policies different from those used for main memory, and a policy must be devised for deciding when to move processes between main memory and secondary storage.

Processes and Memory

Each process operates on a virtual machine that is defined by the architecture of the underlying hardware on which it executes. We are interested in only those machines that include the notion of a virtual address space. A virtual address space is a range of memory locations that a process references independently of the physical memory present in the system. In other words, the virtual address space of a process is independent of the physical address space of the CPU. For a machine to support virtual memory, we also require that the whole of a process's virtual address space does not need to be resident in main memory for that process to execute.

References to the virtual address space—virtual addresses—are translated by hardware into references to physical memory. This operation, termed address translation, permits programs to be loaded into memory at any location without requiring position-dependent addresses in the program to be changed. This relocation of position-dependent addressing is possible because the addresses known to the program do not change. Address translation and virtual addressing are also important in efficient sharing of a CPU, because position independence usually permits context switching to be done quickly.

Most machine architectures provide a contiguous virtual address space for processes. Some machine architectures, however, choose to partition visibly a process's virtual address space into regions termed segments [Intel, 1984]. Such segments usually must be physically contiguous in main memory and must begin at fixed addresses. We shall be concerned with only those systems that do not visibly segment their virtual address space. This use of the word segment is not the same as its earlier use in Section 3.5, when we were describing FreeBSD process segments, such as text and data segments.

When multiple processes are coresident in main memory, we must protect the physical memory associated with each process's virtual address space to ensure that one process cannot alter the contents of another process's virtual address space. This protection is implemented in hardware and is usually tightly coupled with the implementation of address translation. Consequently, the two operations usually are defined and implemented together as hardware termed the memory-management unit.

Virtual memory can be implemented in many ways, some of which are software based, such as overlays. Most effective virtual-memory schemes are, however, hardware based. In these schemes, the virtual address space is divided into fixed-sized units, termed pages, as shown in Figure 5.2. Virtual-memory references are resolved by the address-translation unit to a page in main memory and an offset within that page. Hardware protection is applied by the memory-management unit on a page-by-page basis.

Figure 5.2. Paged virtual-memory scheme. Key: MMU—memory-management unit.

Some systems provide a two-tiered virtual-memory system in which pages are grouped into segments [Organick, 1975]. In these systems, protection is usually at the segment level. In the remainder of this chapter, we shall be concerned with only those virtual-memory systems that are page based.

Paging

Address translation provides the implementation of virtual memory by decoupling the virtual address space of a process from the physical address space of the CPU. Each page of virtual memory is marked as resident or nonresident in main memory. If a process references a location in virtual memory that is not resident, a hardware trap termed a page fault is generated. The servicing of page faults, or paging, permits processes to execute even if they are only partially resident in main memory.

Coffman & Denning [1973] characterize paging systems by three important policies:

  1. When the system loads pages into memory—the fetch policy

  2. Where the system places pages in memory—the placement policy

  3. How the system selects pages to be removed from main memory when pages are unavailable for a placement request—the replacement policy

The performance of modern computers is heavily dependent on one or more highspeed caches to reduce the need to access the much slower main memory. The placement policy should ensure that contiguous pages in virtual memory make the best use of the processor cache. FreeBSD uses a coloring algorithm to ensure good placement. Under a pure demand-paging system, a demand-fetch policy is used, in which only the missing page is fetched, and replacements occur only when main memory is full. Consequently, the performance of a pure demand-paging system depends on only the system's replacement policy. In practice, paging systems do not implement a pure demand-paging algorithm. Instead, the fetch policy often is altered to do prepaging—fetching pages of memory other than the one that caused the page fault—and the replacement policy is invoked before main memory is full.

Replacement Algorithms

The replacement policy is the most critical aspect of any paging system. There is a wide range of algorithms from which we can select in designing a replacement strategy for a paging system. Much research has been carried out in evaluating the performance of different page-replacement algorithms [Bansal & Modha, 2004; Belady, 1966; King, 1971; Marshall, 1979].

A process's paging behavior for a given input is described in terms of the pages referenced over the time of the process's execution. This sequence of pages, termed a reference string, represents the behavior of the process at discrete times during the process's lifetime. Corresponding to the sampled references that constitute a process's reference string are real-time values that reflect whether the associated references resulted in a page fault. A useful measure of a process's behavior is the fault rate, which is the number of page faults encountered during processing of a reference string, normalized by the length of the reference string.

Page-replacement algorithms typically are evaluated in terms of their effectiveness on reference strings that have been collected from execution of real programs. Formal analysis can also be used, although it is difficult to do unless many restrictions are applied to the execution environment. The most common metric used in measuring the effectiveness of a page-replacement algorithm is the fault rate.

Page-replacement algorithms are defined by the criteria that they use for selecting pages to be reclaimed. For example, the optimal replacement policy [Denning, 1970] states that the "best" choice of a page to replace is the one with the longest expected time until its next reference. Clearly, this policy is not applicable to dynamic systems, as it requires a priori knowledge of the paging characteristics of a process. The policy is useful for evaluation purposes, however, as it provides a yardstick for comparing the performance of other page-replacement algorithms.

Practical page-replacement algorithms require a certain amount of state information that the system uses in selecting replacement pages. This state typically includes the reference pattern of a process, sampled at discrete time intervals. On some systems, this information can be expensive to collect [Babaoglu & Joy, 1981]. As a result, the "best" page-replacement algorithm may not be the most efficient.

Working-Set Model

The working-set model assumes that processes exhibit a slowly changing locality of reference. For a period of time, a process operates in a set of subroutines or loops, causing all its memory references to refer to a fixed subset of its address space, termed the working set. The process periodically changes its working set, abandoning certain areas of memory and beginning to access new ones. After a period of transition, the process defines a new set of pages as its working set. In general, if the system can provide the process with enough pages to hold that process's working set, the process will experience a low page-fault rate. If the system cannot provide the process with enough pages for the working set, the process will run slowly and will have a high page-fault rate.

Precise calculation of the working set of a process is impossible without a priori knowledge of that process's memory-reference pattern. However, the working set can be approximated by various means. One method of approximation is to track the number of pages held by a process and that process's page-fault rate. If the page-fault rate increases above a high watermark, the working set is assumed to have increased, and the number of pages held by the process is allowed to grow. Conversely, if the page-fault rate drops below a low watermark, the working set is assumed to have decreased, and the number of pages held by the process is reduced.

Swapping

Swapping is the term used to describe a memory-management policy in which entire processes are moved to and from secondary storage when main memory is in short supply. Swap-based memory-management systems usually are less complicated than are demand-paged systems, since there is less bookkeeping to do. However, pure swapping systems typically are less effective than are paging systems, since the degree of multiprogramming is lowered by the requirement that processes be fully resident to execute. Swapping is sometimes combined with paging in a two-tiered scheme, whereby paging satisfies memory demands until a severe memory shortfall requires drastic action, in which case swapping is used.

In this chapter, a portion of secondary storage that is used for paging or swapping is termed a swap area or swap space. The hardware devices on which these areas reside are termed swap devices.

Advantages of Virtual Memory

There are several advantages to the use of virtual memory on computers capable of supporting this facility properly. Virtual memory allows large programs to be run on machines with main-memory configurations that are smaller than the program size. On machines with a moderate amount of memory, it allows more programs to be resident in main memory to compete for CPU time, as the programs do not need to be completely resident. When programs use sections of their program or data space for some time, leaving other sections unused, the unused sections do not need to be present. Also, the use of virtual memory allows programs to start up faster, since they generally require only a small section to be loaded before they begin processing arguments and determining what actions to take. Other parts of a program may not be needed at all during individual runs. As a program runs, additional sections of its program and data spaces are paged in on demand (demand paging). Finally, there are many algorithms that are more easily programmed by sparse use of a large address space than by careful packing of data structures into a small area. Such techniques are too expensive for use without virtual memory, but they may run much faster when that facility is available, without using an inordinate amount of physical memory.

On the other hand, the use of virtual memory can degrade performance. It is more efficient to load a program all at one time than to load it entirely in small sections on demand. There is a finite cost for each operation, including saving and restoring state and determining which page must be loaded, so some systems use demand paging for only those programs that are larger than some minimum size.

Hardware Requirements for Virtual Memory

Nearly all versions of UNIX have required some form of memory-management hardware to support transparent multiprogramming. To protect processes from modification by other processes, the memory-management hardware must prevent programs from changing their own address mapping. The FreeBSD kernel runs in a privileged mode (kernel mode or system mode) in which memory mapping can be controlled, whereas processes run in an unprivileged mode (user mode). There are several additional architectural requirements for support of virtual memory. The CPU must distinguish between resident and nonresident portions of the address space, must suspend programs when they refer to nonresident addresses, and must resume programs' operation once the operating system has placed the required section in memory. Because the CPU may discover missing data at various times during the execution of an instruction, it must provide a mechanism to save the machine state so that the instruction can be continued or restarted later. The CPU may implement restarting by saving enough state when an instruction begins that the state can be restored when a fault is discovered. Alternatively, instructions could delay any modifications or side effects until after any faults would be discovered so that the instruction execution does not need to back up before restarting. On some computers, instruction backup requires the assistance of the operating system.

Most machines designed to support demand-paged virtual memory include hardware support for the collection of information on program references to memory. When the system selects a page for replacement, it must save the contents of that page if they have been modified since the page was brought into memory. The hardware usually maintains a per-page flag showing whether the page has been modified. Many machines also include a flag recording any access to a page for use by the replacement algorithm.


5.2. Overview of the FreeBSD Virtual-Memory System

The FreeBSD virtual-memory system is based on the Mach 2.0 virtual-memory system [Tevanian, 1987], with updates from Mach 2.5 and Mach 3.0. The Mach virtual-memory system was adopted because it features efficient support for sharing and a clean separation of machine-independent and machine-dependent features, as well as multiprocessor support. None of the original Mach system-call interface remains. It has been replaced with the interface first proposed for 4.2BSD that has been widely adopted by the UNIX industry; the FreeBSD interface is described in Section 5.5.

The virtual-memory system implements protected address spaces into which can be mapped data sources (objects) such as files or private, anonymous pieces of swap space. Physical memory is used as a cache of recently used pages from these objects and is managed by a global page-replacement algorithm.

The virtual address space of most architectures is divided into two parts. For FreeBSD running on 32-bit architectures, the top 1 Gbyte of the address space is reserved for use by the kernel. Systems with many small processes making heavy use of kernel facilities such as networking can be configured to use the top 2 Gbyte for the kernel. The remaining address space is available for use by processes. A traditional UNIX layout is shown in Figure 5.3 (on page 142). Here, the kernel and its associated data structures reside at the top of the address space. The initial text and data areas of the user process start at or near the beginning of memory. Typically, the first 4 or 8 Kbyte of memory are kept off-limits to the process. The reason for this restriction is to ease program debugging; indirecting through a null pointer will cause an invalid address fault instead of reading or writing the program text. Memory allocations made by the running process using the malloc() library routine (or the sbrk system call) are done on the heap that starts immediately following the data area and grows to higher addresses. The argument vector and environment vectors are at the top of the user portion of the address space. The user's stack starts just below these vectors and grows to lower addresses. Subject to only administrative limits, the stack and heap can each grow until they meet. At that point, a process running on a 32-bit machine will be using nearly 3 Gbyte of address space.

Figure 5.3. Layout of virtual address space.

In FreeBSD and other modern UNIX systems that support the mmap system call, address-space usage is less structured. Shared library implementations may place text or data arbitrarily, rendering the notion of predefined regions obsolete. For compatibility, FreeBSD still supports the sbrk call that malloc() uses to provide a contiguous heap region, and the stack region is automatically grown to its size limit. By default, shared libraries are placed just below the run-time configured maximum stack area.

At any time, the currently executing process is mapped into the virtual address space. When the system decides to context switch to another process, it must save the information about the current-process address mapping, then load the address mapping for the new process to be run. The details of this address-map switching are architecture dependent. Some architectures need to change only a few memory-mapping registers that point to the base, and to give the length of memory-resident page tables. Other architectures store the page-table descriptors in special high-speed static RAM. Switching these maps may require dumping and reloading hundreds of map entries.

Both the kernel and user processes use the same basic data structures for the management of their virtual memory. The data structures used to manage virtual memory are as follows:

vmspace | Structure that encompasses both the machine-dependent and machine-independent structures describing a process's address space
vm_map | Highest-level data structure that describes the machine-independent virtual address space
vm_map_entry | Structure that describes a virtually contiguous range of addresses that share protection and inheritance attributes and that use the same backing-store object
object | Structure that describes a source of data for a range of addresses
shadow object | Special object that represents modified copy of original data
vm_page | The lowest-level data structure that represents the physical memory being used by the virtual-memory system

In the remainder of this section, we describe briefly how all these data structures fit together. The remainder of this chapter describes the details of the structures and how the structures are used.

Figure 5.4 shows a typical process address space and associated data structures. The vmspace structure encapsulates the virtual-memory state of a particular process, including the machine-dependent and machine-independent data structures, as well as statistics. The machine-dependent pmap structure is opaque to all but the lowest level of the system and contains all information necessary to manage the memory-management hardware. This pmap layer is the subject of Section 5.13 and is ignored for the remainder of the current discussion. The machine-independent data structures include the address space that is represented by a vm_map structure. The vm_map contains a linked list of vm_map_entry structures, hints for speeding up lookups during memory allocation and page-fault handling, and a pointer to the associated machine-dependent pmap structure contained in the vmspace. A vm_map_entry structure describes a virtually contiguous range of addresses that have the same protection and inheritance attributes. Every vm_map_entry points to a chain of vm_object structures that describes sources of data (objects) that are mapped at the indicated address range. At the tail of the chain is the original mapped data object, usually representing a persistent data source, such as a file. Interposed between that object and the map entry are one or more transient shadow objects that represent modified copies of the original data. These shadow objects are discussed in detail in Section 5.5.

Figure 5.4. Data structures that describe a process address space.

[View full size image]

Each vm_object structure contains a linked list of vm_page structures representing the physical-memory cache of the object, as well as the type of and a pointer to the pager structure that contains information on how to page in or page out data from its backing store. There is a vm_page structure allocated for every page of physical memory managed by the virtual-memory system, where a page here may be a collection of multiple, contiguous hardware pages that will be treated by the machine-dependent layer as though they were a single unit. The structure also contains the status of the page (e.g., modified or referenced) and links for various paging queues. A vm_page structure is most commonly and quickly found through a global hash list that is keyed by the object and logical page within that object whose contents are held by the page.

All structures contain the necessary interlocks for multithreading in a multiprocessor environment. The locking is fine grained, with at least one lock per instance of a data structure. Many of the structures contain multiple locks to protect individual fields.


5.3. Kernel Memory Management

There are two ways in which the kernel's memory can be organized. The most common is for the kernel to be permanently mapped into the high part of every process address space. In this model, switching from one process to another does not affect the kernel portion of the address space. The alternative organization is to switch between having the kernel occupy the whole address space and mapping the currently running process into the address space. Having the kernel permanently mapped does reduce the amount of address space available to a large process (and the kernel), but it also reduces the cost of data copying. Many system calls require data to be transferred between the currently running user process and the kernel. With the kernel permanently mapped, the data can be copied via the efficient block-copy instructions. If the kernel is alternately mapped with the process, data copying requires the use of special instructions that copy to and from the previously mapped address space. These instructions are usually a factor of 2 slower than the standard block-copy instructions. Since up to one-third of the kernel time is spent in copying between the kernel and user processes, slowing this operation by a factor of 2 significantly slows system throughput.

Although the kernel is able to freely read and write the address space of the user process, the converse is not true. The kernel's range of virtual address space is marked inaccessible to all user processes. Writing is restricted so user processes cannot tamper with the kernel's data structures. Reading is restricted so user processes cannot watch sensitive kernel data structures, such as the terminal input queues, that include such things as users typing their passwords.

Usually, the hardware dictates which organization can be used. All the architectures supported by FreeBSD map the kernel into the top of the address space.

Kernel Maps and Submaps

When the system boots, the first task that the kernel must do is to set up data structures to describe and manage its address space. Like any process, the kernel has a vm_map with a corresponding set of vm_map_entry structures that describe the use of a range of addresses. Submaps are a special kernel-only construct used to isolate and constrain address-space allocation for kernel subsystems. One use is in subsystems that require contiguous pieces of the kernel address space. To avoid intermixing of unrelated allocations within an address range, that range is covered by a submap, and only the appropriate subsystem can allocate from that map. For example, several network buffer (mbuf) manipulation macros use address arithmetic to generate unique indices, thus requiring the network buffer region to be contiguous. Parts of the kernel may also require addresses with particular alignments or even specific addresses. Both can be ensured by use of submaps. Finally, submaps can be used to limit statically the amount of address space and hence the physical memory consumed by a subsystem.

A typical layout of the kernel map is shown in Figure 5.5. The kernel's address space is described by the vm_map structure shown in the upperleft corner of the figure. Pieces of the address space are described by the vm_map_entry structures that are linked in ascending address order from K0 to K8 on the vm_map structure. Here, the kernel text, initialized data, uninitialized data, and initially allocated data structures reside in the range K0 to Kl and are represented by the first vm_map_entry. The next vm_map_entry is associated with the address range from K2 to K6. This piece of the kernel address space is being managed via a submap headed by the referenced vm_map structure. This submap currently has two parts of its address space used: the address range K2 to K3, and the address range K4 to K5. These two submaps represent the kernel malloc arena and the network buffer arena, respectively. The final part of the kernel address space is being managed in the kernel's main map, the address range K7 to K8 representing the kernel I/O staging area.

Figure 5.5. Kernel address-space maps.

Kernel Address-Space Allocation

The virtual-memory system implements a set of primitive functions for allocating and freeing the page-aligned, page-rounded virtual-memory ranges that the kernel uses. These ranges may be allocated either from the main kernel-address map or from a submap. The allocation routines take a map and size as parameters but do not take an address. Thus, specific addresses within a map cannot be selected. There are different allocation routines for obtaining nonpageable and pageable memory ranges.

A nonpageable, or wired, range has physical memory assigned at the time of the call, and this memory is not subject to replacement by the pageout daemon. Wired pages must never cause a page fault that might result in a blocking operation. Wired memory is allocated with kmem_alloc() and kmem_malloc(). Kmem_alloc() returns zero-filled memory and may block if insufficient physical memory is available to honor the request. It will return a failure only if no address space is available in the indicated map. Kmem_malloc() is a variant of kmem_alloc() used by only the general allocator, malloc(), described in the next subsection. This routine has a nonblocking option that protects callers against inadvertently blocking on kernel data structures; it will fail if insufficient physical memory is available to fill the requested range. This nonblocking option allocates memory at interrupt time and during other critical sections of code. In general, wired memory should be allocated via the general-purpose kernel allocator. Kmem_alloc() should be used only to allocate memory from specific kernel submaps.

Pageable kernel virtual memory can be allocated with kmem_alloc_pageable() and kmem_alloc_wait(). A pageable range has physical memory allocated on demand, and this memory can be written out to backing store by the pageout daemon as part of the latter's normal replacement policy. Kmem_alloc_pageable() will return an error if insufficient address space is available for the desired allocation; kmem_alloc_wait() will block until space is available. Currently, pageable kernel memory is used only for temporary storage of exec arguments.

Kmem_free() deallocates kernel wired memory and pageable memory allocated with kmem_alloc_pageable(). Kmem_free_wakeup() should be used with kmem_alloc_wait() because it wakes up any processes waiting for address space in the specified map.

Kernel Malloc

The kernel also provides a generalized nonpageable memory-allocation and freeing mechanism that can handle requests with arbitrary alignment or size, as well as allocate memory at interrupt time. So it is the preferred way to allocate kernel memory other than large, fixed-size structures that are better handled by the zone allocator described in the next subsection. This mechanism has an interface similar to that of the well-known memory allocator provided for applications programmers through the C library routines malloc() and free(). Like the C library interface, the allocation routine takes a parameter specifying the size of memory that is needed. The range of sizes for memory requests are not constrained. The free routine takes a pointer to the storage being freed, but it does not require the size of the piece of memory being freed.

Often, the kernel needs a memory allocation for the duration of a single system call. In a user process, such short-term memory would be allocated on the run-time stack. Because the kernel has a limited run-time stack, it is not feasible to allocate even moderate blocks of memory on it. Consequently, such memory must be allocated dynamically. For example, when the system must translate a pathname, it must allocate a 1-Kbyte buffer to hold the name. Other blocks of memory must be more persistent than a single system call and have to be allocated from dynamic memory. Examples include protocol control blocks that remain throughout the duration of a network connection.

The design specification for a kernel memory allocator is similar, but not identical, to the design criteria for a user-level memory allocator. One criterion for a memory allocator is that it make good use of the physical memory. Use of memory is measured by the amount of memory needed to hold a set of allocations at any point in time. Percentage utilization is expressed as

Here, requested is the sum of the memory that has been requested and not yet freed; required is the amount of memory that has been allocated for the pool from which the requests are filled. An allocator requires more memory than requested because of fragmentation and a need to have a ready supply of free memory for future requests. A perfect memory allocator would have a utilization of 100 percent. In practice, a 50 percent utilization is considered good [Korn & Vo, 1985].

Good memory utilization in the kernel is more important than in user processes. Because user processes run in virtual memory, unused parts of their address space can be paged out. Thus, pages in the process address space that are part of the required pool that are not being requested do not need to tie up physical memory. Since the kernel malloc arena is not paged, all pages in the required pool are held by the kernel and cannot be used for other purposes. To keep the kernel-utilization percentage as high as possible, the kernel should release unused memory in the required pool rather than hold it, as is typically done with user processes. Because the kernel can manipulate its own page maps directly, freeing unused memory is fast; a user process must do a system call to free memory.

The most important criterion for a kernel memory allocator is that it be fast. A slow memory allocator will degrade the system performance because memory allocation is done frequently. Speed of allocation is more critical when executing in the kernel than it is in user code because the kernel must allocate many data structures that user processes can allocate cheaply on their run-time stack. In addition, the kernel represents the platform on which all user processes run, and if it is slow, it will degrade the performance of every process that is running.

Another problem with a slow memory allocator is that programmers of frequently used kernel interfaces will think that they cannot afford to use the memory allocator as their primary one. Instead, they will build their own memory allocator on top of the original by maintaining their own pool of memory blocks. Multiple allocators reduce the efficiency with which memory is used. The kernel ends up with many different free lists of memory instead of a single free list from which all allocations can be drawn. For example, consider the case of two subsystems that need memory. If they have their own free lists, the amount of memory tied up in the two lists will be the sum of the greatest amount of memory that each of the two subsystems has ever used. If they share a free list, the amount of memory tied up in the free list may be as low as the greatest amount of memory that either subsystem used. As the number of subsystems grows, the savings from having a single free list grow.

The kernel memory allocator uses a hybrid strategy. Small allocations are done using a power-of-2 list strategy. Using the zone allocator (described in the next subsection), the kernel creates a set of zones with one for each power-of-two between 16 and the page size. The allocation simply requests a block of memory from the appropriate zone. Usually, the zone will have an available piece of memory that it can return. Only if every piece of memory in the zone is in use will the zone allocator have to do a full allocation. When forced to do an additional allocation, it allocates an entire page and carves it up into the appropriate-sized pieces. This strategy speeds future allocations because several pieces of memory become available as a result of the call into the allocator.

Freeing a small block also is fast. The memory is simply returned to the zone from which it came.

Because of the inefficiency of power-of-2 allocation strategies for large allocations, the allocation method for large blocks is based on allocating pieces of memory in multiples of pages. The algorithm switches to the slower but more memory-efficient strategy for allocation sizes larger than a page. This value is chosen because the power-of-2 algorithm yields sizes of 1, 2, 4, 8, . . ., n pages, whereas the large block algorithm that allocates in multiples of pages yields sizes of 1, 2, 3, 4,. . ., n pages. Thus, for allocations of greater than one page, the large block algorithm will use less than or equal to the number of pages used by the power-of-2 algorithm, so the threshold between the large and small allocators is set at one page.

Large allocations are first rounded up to be a multiple of the page size. The allocator then uses a "first-fit" algorithm to find space in the kernel address arena set aside for dynamic allocations. On a machine with a 4-Kbyte page size, a request for a 20-Kbyte piece of memory will use exactly five pages of memory, rather than the eight pages used with the power-of-2 allocation strategy. When a large piece of memory is freed, the memory pages are returned to the free-memory pool and the vm_map_entry structure is deleted from the submap, effectively coalescing the freed piece with any adjacent free space.

Because the size is not specified when a block of memory is freed, the allocator must keep track of the sizes of the pieces that it has handed out. Many allocators increase the allocation request by a few bytes to create space to store the size of the block in a header just before the allocation. However, this strategy doubles the memory requirement for allocations that request a power-of-2-sized block. Therefore, instead of storing the size of each piece of memory with the piece itself, the zone allocator associates the size information with the memory page. Locating the allocation size outside of the allocated block improved utilization far more than expected. The reason is that many allocations in the kernel are for blocks of memory whose size is exactly a power of 2. The size of these requests would be nearly doubled if the more typical strategy were used. Now they can be accommodated with no wasted memory.

The allocator can be called both from the top half of the kernel that is willing to wait for memory to become available and from the interrupt routines in the bottom half of the kernel that cannot wait for memory to become available. Clients show their willingness (and ability) to wait with a flag to the allocation routine. For clients that are willing to wait, the allocator guarantees that their request will succeed. Thus, these clients do not need to check the return value from the allocator. If memory is unavailable and the client cannot wait, the allocator returns a null pointer. These clients must be prepared to cope with this (typically infrequent) condition (usually by giving up and hoping to succeed later). The details of the kernel memory allocator are further described in McKusick & Karels [1988].

Kernel Zone Allocator

Some commonly allocated items in the kernel such as process, thread, vnode, and control-block structures are not well handled by the general purpose malloc interface. These structures share several characteristics:

  • They tend to be large and hence wasteful of space. For example, the process structure is about 550 bytes, which when rounded up to a power-of-2 size requires 1024 bytes of memory.

  • They tend to be common. Because they are individually wasteful of space, collectively they waste too much space compared to a denser representation.

  • They are often linked together in long lists. If the allocation of each structure begins on a page boundary, then the list pointers will all be at the same offset from the beginning of the page. When traversing these structures, the linkage pointers will all be located on the same hardware cache line, causing each step along the list to produce a cache miss, making the list-traversal slow.

  • These structures often contain many lists and locks that must be initialized before use. If there is a dedicated pool of memory for each structure, then these substructures need to be initialized only when the pool is first created rather than after every allocation.

For these reasons, FreeBSD provides a zone allocator (sometimes referred to as a slab allocator). The zone allocator provides an efficient interface for managing a collection of items of the same size and type [Bonwick, 1994; Bonwick & Adams, 2001]. The creation of separate zones runs counter to the desire to keep all memory in a single pool to maximize utilization efficiency. However, the benefits from segregating memory for the small set of structures for which the zone allocator is appropriate outweighs the efficiency gains from keeping them in the general pool. The zone allocator minimizes the impact of the separate pools by freeing memory from a zone based on a reduction in demand for objects from the zone and when notified of a memory shortage by the pageout daemon.

A zone is an extensible collection of items of identical size. The zone allocator keeps track of the active and free items and provides functions for allocating items from the zone and for releasing them back to make them available for later use. Memory is allocated to the zone as necessary.

Items in the pool are type stable. The memory in the pool will not be used for any other purpose. A structure in the pool need only be initialized on its first use. Later uses may assume that the initialized values will retain their contents as of the previous free. A callback is provided before memory is freed from the zone to allow any persistent state to be cleaned up.

A new zone is created with the uma_zcreate() function. It must specify the size of the items to be allocated and register two sets of functions. The first set is called whenever an item is allocated or freed from the zone. They typically track the number of allocated items. The second set is called whenever memory is allocated or freed from the zone. They provide hooks for initializing or destroying substructures within the items such as locks or mutexes when they are first allocated or are about to be freed.

Items are allocated with uma_zalloc(), which takes a zone identifier returned by uma_zcreate(). Items are freed with uma_zfree(), which takes a zone identifier and a pointer to the item to be freed. No size is needed when allocating or freeing, since the item size was set when the zone was created.

To aid performance on multiprocessor systems, a zone may be backed by pages from the per-CPU caches of each of the processors for objects that are likely to have processor affinity. For example, if a particular process is likely to be run on the same CPU, then it would be sensible to allocate its structure from the cache for that CPU.

The zone allocator provides the uma_zone_set_max() function to set the upper limit of items in the zone. The limit on the total number of items in the zone includes the allocated and free items, including the items in the per-CPU caches. On multiprocessor systems, it may not be possible to allocate a new item for a particular CPU because the limit has been hit and all the free items are in the caches of the other CPUs.


5.4. Per-Process Resources

As we have already seen, a process requires a process entry and a kernel stack. The next major resource that must be allocated is its virtual memory. The initial virtual-memory requirements are defined by the header in the process's executable. These requirements include the space needed for the program text, the initialized data, the uninitialized data, and the run-time stack. During the initial startup of the program, the kernel will build the data structures necessary to describe these four areas. Most programs need to allocate additional memory. The kernel typically provides this additional memory by expanding the uninitialized data area.

Most FreeBSD programs use shared libraries. The header for the executable will describe the libraries that it needs (usually the C library, and possibly others). The kernel is not responsible for locating and mapping these libraries during the initial execution of the program. Finding, mapping, and creating the dynamic linkages to these libraries is handled by the user-level startup code prepended to the file being executed. This startup code usually runs before control is passed to the main entry point of the program [Gingell et al., 1987].

FreeBSD Process Virtual-Address Space

The initial layout of the address space for a process is shown in Figure 5.6 (on page 152). As discussed in Section 5.2, the address space for a process is described by that process's vmspace structure. The contents of the address space are defined by a list of vm_map_entry structures, each structure describing a region of virtual address space that resides between a start and an end address. A region describes a range of memory that is being treated in the same way. For example, the text of a program is a region that is read-only and is demand paged from the file on disk that contains it. Thus, the vm_map_entry also contains the protection mode to be applied to the region that it describes. Each vm_map_entry structure also has a pointer to the object that provides the initial data for the region. The object also stores the modified contents either transiently, when memory is being reclaimed, or more permanently, when the region is no longer needed. Finally, each vm_map_entry structure has an offset that describes where within the object the mapping begins.

Figure 5.6. Layout of an address space.

The example shown in Figure 5.6 represents a process just after it has started execution. The first two map entries both point to the same object; here, that object is the executable. The executable consists of two parts: the text of the program that resides at the beginning of the file and the initialized data area that follows at the end of the text. Thus, the first vm_map_entry describes a read-only region that maps the text of the program. The second vm_map_entry describes the copy-on-write region that maps the initialized data of the program that follows the program text in the file (copy-on-write is described in Section 5.6). The offset field in the entry reflects this different starting location. The third and fourth vm_map_entry structures describe the uninitialized data and stack areas, respectively. Both of these areas are represented by anonymous objects. An anonymous object provides a zero-filled page on first use and arranges to store modified pages in the swap area if memory becomes tight. Anonymous objects are described in more detail later in this section.

Page-Fault Dispatch

When a process attempts to access a piece of its address space that is not currently resident, a page fault occurs. The page-fault handler in the kernel is presented with the virtual address that caused the fault and the type of access that was attempted (read or write). The fault is handled with the following four steps:

1. Find the vmspace structure for the faulting process; from that structure, find the head of the vm_map_entry list.

2. Traverse the vm_map_entry list, starting at the entry indicated by the map hint; for each entry, check whether the faulting address falls within its start and end address range. If the kernel reaches the end of the list without finding any valid region, the faulting address is not within any valid part of the address space for the process, so send the process a segment fault signal.

3. Having found a vm_map_entry that contains the faulting address, convert that address to an offset within the underlying object. Calculate the offset within the object as

object_offset = fault_address
    - vm_map_entry  4start_address
    + vm_map_entry  object_offset

Subtract off the start address to give the offset into the region mapped by the vm_map_entry. Add in the object offset to give the absolute offset of the page within the object.

4. Present the absolute object offset to the underlying object, which allocates a vm_page structure and uses its pager to fill the page. The object then returns a pointer to the vm_page structure, which is mapped into the faulting location in the process address space.

Once the appropriate page has been mapped into the faulting location, the page-fault handler returns and reexecutes the faulting instruction.

Mapping to Objects

Objects are used to hold information about either a file or about an area of anonymous memory. Whether a file is mapped by a single process in the system or by many processes in the system, it will always be represented by a single object. Thus, the object is responsible for maintaining all the state about those pages of a file that are resident. All references to that file will be described by vm_map_entry structures that reference the same object. An object never stores the same page of a file in more than one physical-memory page, so all mappings will get a consistent view of the file.

An object stores the following information:

  • A list of the pages for that object that are currently resident in main memory; a page may be mapped into multiple address spaces, but it is always claimed by exactly one object

  • A count of the number of vm_map_entry structures or other objects that reference the object

  • The size of the file or anonymous area described by the object

  • The number of memory-resident pages held by the object

  • Pointer to shadow objects (described in Section 5.5)

  • The type of pager for the object; the pager is responsible for providing the data to fill a page and for providing a place to store the page when it has been modified (pagers are covered in Section 5.10)

There are three types of objects in the system:

  • Named objects represent files; they may also represent hardware devices that are able to provide mapped memory such as frame buffers.

  • Anonymous objects represent areas of memory that are zero filled on first use; they are abandoned when they are no longer needed.

  • Shadow objects hold private copies of pages that have been modified; they are abandoned when they are no longer referenced.

These objects are often referred to as "internal" objects in the source code. The type of an object is defined by the type of pager that that object uses to fulfill page-fault requests.

A named object uses either the device pager, if it maps a hardware device, or the vnode pager, if it is backed by a file in the filesystem. The device pager services a page fault by returning the appropriate address for the device being mapped. Since the device memory is separate from the main memory on the machine, it will never be selected by the pageout daemon. Thus, the device pager never has to handle a pageout request.

The vnode pager provides an interface to objects that represent files in the filesystem. The vnode pager keeps a reference to a vnode that represents the file being mapped in the object. The vnode pager services a pagein request by doing a read on the vnode; it services a pageout request by doing a write to the vnode. Thus, the file itself stores the modified pages. In cases where it is not appropriate to modify the file directly, such as an executable that does not want to modify its initialized data pages, the kernel must interpose an anonymous shadow object between the vm_map_entry and the object representing the file; see Section 5.5.

Anonymous objects use the swap pager. An anonymous object services pagein requests by getting a page of memory from the free list and zeroing that page. When a pageout request is made for a page for the first time, the swap pager is responsible for finding an unused page in the swap area, writing the contents of the page to that space, and recording where that page is stored. If a pagein request comes for a page that had been previously paged out, the swap pager is responsible for finding where it stored that page and reading back the contents into a free page in memory. A later pageout request for that page will cause the page to be written out to the previously allocated location.

Shadow objects also use the swap pager. They work just like anonymous objects, except that the swap pager provides their initial pages by copying existing pages in response to copy-on-write faults instead of by zero-filling pages.

Further details on the pagers are given in Section 5.10.

Objects

Each virtual-memory object has a pager type, pager handle, and pager private data associated with it. Objects that map files have a vnode-pager type associated with them. The handle for the vnode-pager type is a pointer to the vnode on which to do the I/O, and the private data is the size of the vnode at the time that the mapping was done. Every vnode that maps a file has an object associated with it. When a fault occurs for a file that is mapped into memory, the object associated with the file can be checked to see whether the faulted page is resident. If the page is resident, it can be used. If the page is not resident, a new page is allocated, and the vnode pager is requested to fill the new page.

Caching in the virtual-memory system is identified by an object that is associated with a file or region that it represents. Each object contains pages that are the cached contents of its associated file or region. Objects are reclaimed as soon as their reference count drops to zero. Pages associated with reclaimed objects are moved to the free list. Objects that represent anonymous memory are reclaimed as part of cleaning up a process as it exits. However, objects that refer to files are persistent. When the reference count on a vnode drops to zero, it is stored on a least-recently used (LRU) list known as the vnode cache; see Section 6.6. The vnode does not release its object until the vnode is reclaimed and reused for another file. Unless there is pressure on the memory, the object associated with the vnode will retain its pages. If the vnode is reactivated and a page fault occurs before the associated page is freed, that page can be reattached rather than being reread from disk.

This cache is similar to the text cache found in earlier versions of BSD in that it provides performance improvements for short-running but frequently executed programs. Frequently executed programs include those used to list the contents of directories, show system status, or perform the intermediate steps involved in compiling a program. For example, consider a typical application that is made up of multiple source files. Each of several compiler steps must be run on each file in turn. The first time that the compiler is run, the executable files associated with its various components are read in from the disk. For each file compiled thereafter, the previously created executable files are found, as well as any previously read header files, alleviating the need to reload them from disk each time.

Objects to Pages

When the system is first booted, the kernel looks through the physical memory on the machine to find out how many pages are available. After the physical memory that will be dedicated to the kernel itself has been deducted, all the remaining pages of physical memory are described by vm_page structures. These vm_page structures are all initially placed on the memory free list. As the system starts running and processes begin to execute, they generate page faults. Each page fault is matched to the object that covers the faulting piece of address space. The first time that a piece of an object is faulted, it must allocate a page from the free list and must initialize that page either by zero-filling it or by reading its contents from the filesystem. That page then becomes associated with the object. Thus, each object has its current set of vm_page structures linked to it. A page can be associated with at most one object at a time. Although a file may be mapped into several processes at once, all those mappings reference the same object. Having a single object for each file ensures that all processes will reference the same physical pages.

If memory becomes scarce, the paging daemon will search for pages that have not been used actively. Before these pages can be used by a new object, they must be removed from all the processes that currently have them mapped, and any modified contents must be saved by the object that owns them. Once cleaned, the pages can be removed from the object that owns them and can be placed on the free list for reuse. The details of the paging system are described in Section 5.12.


5.5. Shared Memory

In Section 5.4, we explained how the address space of a process is organized. This section shows the additional data structures needed to support shared address space between processes. Traditionally, the address space of each process was completely isolated from the address space of all other processes running on the system. The only exception was read-only sharing of program text. All interprocess communication was done through well-defined channels that passed through the kernel: pipes, sockets, files, and special devices. The benefit of this isolated approach is that, no matter how badly a process destroys its own address space, it cannot affect the address space of any other process running on the system. Each process can precisely control when data are sent or received; it can also precisely identify the locations within its address space that are read or written. The drawback of this approach is that all interprocess communication requires at least two system calls: one from the sending process and one from the receiving process. For high volumes of interprocess communication, especially when small packets of data are being exchanged, the overhead of the system calls dominates the communications cost.

Shared memory provides a way to reduce interprocess-communication costs dramatically. Two or more processes that wish to communicate map the same piece of read-write memory into their address space. Once all the processes have mapped the memory into their address space, any changes to that piece of memory are visible to all the other processes, without any intervention by the kernel. Thus, interprocess communication can be achieved without any system-call overhead other than the cost of the initial mapping. The drawback to this approach is that, if a process that has the memory mapped corrupts the data structures in that memory, all the other processes mapping that memory also are corrupted. In addition, there is the complexity faced by the application developer who must develop data structures to control access to the shared memory and must cope with the race conditions inherent in manipulating and controlling such data structures that are being accessed concurrently.

Some UNIX variants have a kernel-based semaphore mechanism to provide the needed serialization of access to the shared memory. However, both getting and setting such semaphores require system calls. The overhead of using such semaphores is comparable to that of using the traditional interprocess-communication methods. Unfortunately, these semaphores have all the complexity of shared memory, yet confer little of its speed advantage. The primary reason to introduce the complexity of shared memory is for the commensurate speed gain. If this gain is to be obtained, most of the data-structure locking needs to be done in the shared memory segment itself. The kernel-based semaphores should be used for only those rare cases where there is contention for a lock and one process must wait. Consequently, modern interfaces, such as POSIX Pthreads, are designed such that the semaphores can be located in the shared memory region. The common case of setting or clearing an uncontested semaphore can be done by the user process, without calling the kernel. There are two cases where a process must do a system call. If a process tries to set an already locked semaphore, it must call the kernel to block until the semaphore is available. This system call has little effect on performance because the lock is contested, so it is impossible to proceed, and the kernel must be invoked to do a context switch anyway. If a process clears a semaphore that is wanted by another process, it must call the kernel to awaken that process. Since most locks are uncontested, the applications can run at full speed without kernel intervention.

Mmap Model

When two processes wish to create an area of shared memory, they must have some way to name the piece of memory that they wish to share, and they must be able to describe its size and initial contents. The system interface describing an area of shared memory accomplishes all these goals by using files as the basis for describing a shared memory segment. A process creates a shared memory segment by using

caddr_t addr = mmap(
    caddr_t address, /* base address */
    size_t length,   /* length of region */
    int protection,  /* protection of region */
    int flags,       /* mapping flags */
    int fd,          /* file to map */
    off_t offset);   /* offset to begin mapping */

to map the file referenced by descriptor fd starting at file offset offset into its address space starting at addr and continuing for len bytes with access permission prot. The flags parameter allows a process to specify whether it wants to make a shared or private mapping. Changes made to a shared mapping are written back to the file and are visible to other processes. Changes made to a private mapping are not written back to the file and are not visible to other processes. Two processes that wish to share a piece of memory request a shared mapping of the same file into their address space. Thus, the existing and well-understood filesystem name space identifies shared objects. The contents of the file are used as the initial value of the memory segment. All changes made to the mapping are reflected back into the contents of the file, so long-term state can be maintained in the shared memory region, even across invocations of the sharing processes.

Some applications want to use shared memory purely as a short-term interprocess-communication mechanism. They need an area of memory that is initially zeroed and whose contents are abandoned when they are done using it. Such processes neither want to pay the relatively high startup cost associated with paging in the contents of a file to initialize a shared memory segment nor to pay the shutdown costs of writing modified pages back to the file when they are done with the memory. Although FreeBSD does provide the limited and quirky naming scheme of the System V shmem interface as a rendezvous mechanism for such short-term shared memory, the designers ultimately decided that all naming of memory objects for mmap should use the filesystem name space. To provide an efficient mechanism for short-term shared memory, they created a virtual-memory-resident filesystem for transient objects. Unless memory is in high demand, files created in the virtual-memory-resident filesystem reside entirely in memory. Thus, both the initial paging and later write-back costs are eliminated. Typically, a virtual-memory-resident filesystem is mounted on /tmp. Two processes wishing to create a transient area of shared memory create a file in /tmp that they can then both map into their address space.

When a mapping is no longer needed, it can be removed using

munmap(caddr_t address, size_t length);

The munmap system call removes any mappings that exist in the address space, starting at addr and continuing for len bytes. There are no constraints between previous mappings and a later munmap. The specified range may be a subset of a previous mmap, or it may encompass an area that contains many mmap'ed files. When a process exits, the system does an implied munmap over its entire address space.

During its initial mapping, a process can set the protections on a page to allow reading, writing, and/or execution. The process can change these protections later by using

mprotect(caddr_t address, int length, int protection);

This feature can be used by debuggers when they are trying to track down a memory-corruption bug. By disabling writing on the page containing the data structure that is being corrupted, the debugger can trap all writes to the page and verify that they are correct before allowing them to occur.

Traditionally, programming for real-time systems has been done with specially written operating systems. In the interests of reducing the costs of real-time applications and of using the skills of the large body of UNIX programmers, companies developing real-time applications have expressed increased interest in using UNIX-based systems for writing these applications. Two fundamental requirements of a real-time system are guaranteed maximum latencies and predictable execution times. Predictable execution time is difficult to provide in a virtual-memory-based system, since a page fault may occur at any point in the execution of a program, resulting in a potentially large delay while the faulting page is retrieved from the disk or network. To avoid paging delays, the system allows a process to force its pages to be resident, and not paged out, by using

mlock(caddr_t address, size_t length);

As long as the process limits its accesses to the locked area of its address space, it can be sure that it will not be delayed by page faults. To prevent a single process from acquiring all the physical memory on the machine to the detriment of all other processes, the system imposes a resource limit to control the amount of memory that may be locked. Typically, this limit is set to no more than one-third of the physical memory, and it may be set to zero by a system administrator that does not want random processes to be able to monopolize system resources.

When a process has finished with its time-critical use of an mlock'ed region, it can release the pages using

munlock(caddr_t address, size_t length);

After the munlock call, the pages in the specified address range are still accessible, but they may be paged out if memory is needed and they are not accessed.

An application may need to ensure that certain records are committed to disk without forcing the writing of all the dirty pages of a file done by the fsync system call. For example, a database program may want to commit a single piece of metadata without writing back all the dirty blocks in its database file. A process does this selective synchronization using

msync(caddr_t address, int length);

Only those modified pages within the specified address range are written back to the filesystem. The msync system call has no effect on anonymous regions.

Shared Mapping

When multiple processes map the same file into their address space, the system must ensure that all the processes view the same set of memory pages. As shown in Section 5.4, each file that is being used actively by a client of the virtual-memory system is represented by an object. Each mapping that a process has to a piece of a file is described by a vm_map_entry structure. An example of two processes mapping the same file into their address space is shown in Figure 5.7. When a page fault occurs in one of these processes, the process's vm_map_entry references the object to find the appropriate page. Since all mappings reference the same object, the processes will all get references to the same set of physical memory, thus ensuring that changes made by one process will be visible in the address spaces of the other processes as well.

Figure 5.7. Multiple mappings to a file.

Private Mapping

A process may request a private mapping of a file. A private mapping has two main effects:

  1. Changes made to the memory mapping the file are not reflected back into the mapped file.

  2. Changes made to the memory mapping the file are not visible to other processes mapping the file.

An example of the use of a private mapping would be during program debugging. The debugger will request a private mapping of the program text so that, when it sets a breakpoint, the modification is not written back into the executable stored on the disk and is not visible to the other (presumably nondebugging) processes executing the program.

The kernel uses shadow objects to prevent changes made by a process from being reflected back to the underlying object. The use of a shadow object is shown in Figure 5.8. When the initial private mapping is requested, the file object is mapped into the requesting-process address space, with copy-on-write semantics. If the process attempts to write a page of the object, a page fault occurs and traps into the kernel. The kernel makes a copy of the page to be modified and hangs it from the shadow object. In this example, process A has modified page 0 of the file object. The kernel has copied page 0 to the shadow object that is being used to provide the private mapping for process A.

Figure 5.8. Use of a shadow object for a private mapping.

If free memory is limited, it would be better simply to move the modified page from the file object to the shadow object. The move would reduce the immediate demand on the free memory, because a new page would not have to be allocated. The drawback to this optimization is that, if there is a later access to the file object by some other process, the kernel will have to allocate a new page. The kernel will also have to pay the cost of doing an I/O operation to reload the page contents. In FreeBSD, the virtual-memory system never moves the page rather than copying it.

When a page fault for the private mapping occurs, the kernel traverses the list of objects headed by the vm_map_entry, looking for the faulted page. The first object in the chain that has the desired page is the one that is used. If the search gets to the final object on the chain without finding the desired page, then the page is requested from that final object. Thus, pages on a shadow object will be used in preference to the same pages in the file object itself. The details of page-fault handling are given in Section 5.11.

When a process removes a mapping from its address space (either explicitly from an munmap request or implicitly when the address space is freed on process exit), pages held by its shadow object are not written back to the file object. The shadow-object pages are simply placed back on the memory free list for immediate reuse.

When a process forks, it does not want changes to its private mappings made after it forked to be visible in its child; similarly, the child does not want its changes to be visible in its parent. The result is that each process needs to create a shadow object if it continues to make changes in a private mapping. When process A in Figure 5.8 forks, a set of shadow-object chains is created, as shown in Figure 5.9 (on page 162). In this example, process A modified page 0 before it forked and then later modified page 1. Its modified version of page 1 hangs off its new shadow object, so those modifications will not be visible to its child. Similarly, its child has modified page 0. If the child were to modify page 0 in the original shadow object, that change would be visible in its parent. Thus, the child process must make a new copy of page 0 in its own shadow object.

Figure 5.9. Shadow-object chains.

If the system runs short of memory, the kernel may need to reclaim inactive memory held in a shadow object. The kernel assigns to the swap pager the task of backing the shadow object. The swap pager sets up data structures (described in Section 5.10) that can describe the entire contents of the shadow object. It then allocates enough swap space to hold the requested shadow pages and writes them to that area. These pages can then be freed for other uses. If a later page fault requests a swapped-out page, then a new page of memory is allocated and its contents are reloaded with an I/O from the swap area.

Collapsing of Shadow Chains

When a process with a private mapping removes that mapping either explicitly with an munmap system call or implicitly by exiting, its parent or child process may be left with a chain of shadow objects. Usually, these chains of shadow objects can be collapsed into a single shadow object, often freeing up memory as part of the collapse. Consider what happens when process A exits in Figure 5.9. First, shadow object 3 can be freed along with its associated page of memory. This deallocation leaves shadow objects 1 and 2 in a chain with no intervening references. Thus, these two objects can be collapsed into a single shadow object. Since they both contain a copy of page 0, and since only the page 0 in shadow object 2 can be accessed by the remaining child process, the page 0 in shadow object 1 can be freed along with shadow object 1 itself.

If the child of process A were to exit, then shadow object 2 and the associated page of memory could be freed. Shadow objects 1 and 3 would then be in a chain that would be eligible for collapse. Here, there are no common pages, so object 3 would retain its own page 1 and acquire page 0 from shadow object 1. Object 1 would then be freed. In addition to merging the pages from the two objects, the collapse operation requires a similar merger of any swap space that has been allocated by the two objects. If page 2 had been copied to object 3 and page 4 had been copied to object 1, but these pages were later reclaimed, the pager for object 3 would hold a swap block for page 2, and the pager for object 1 would hold a swap block for page 4. Before freeing object 1, its swap block for page 4 would have to be moved over to object 3.

A performance problem can arise if either a process or its children repeatedly fork. Without some intervention, they can create long chains of shadow objects. If the processes are long-lived, the system does not get an opportunity to collapse these shadow-object chains. Traversing these long chains of shadow objects to resolve page faults is time consuming, and many inaccessible pages can build up forcing the system to needlessly page them out to reclaim them.

One alternative would be to calculate the number of live references to a page after each copy-on-write fault. When only one live reference remained, the page could be moved to the shadow object that still referenced it. When all the pages had been moved out of a shadow object, it could be removed from the chain. For example, in Figure 5.9, when the child of process A wrote to page 0, a copy of page 0 was made in shadow object 2. At that point, the only live reference to page 0 in object 1 was from process A. Thus, the page 0 in object 1 could be moved to object 3. That would leave object 1 with no pages, so it could be reclaimed leaving objects 2 and 3 pointing at the file object directly. Unfortunately, this strategy would add considerable overhead to the page-fault handling routine which would noticeably slow the overall performance of the system. So FreeBSD does not make this optimization.

FreeBSD uses a lower-cost heuristic to reduce the length of shadow chains. When taking a copy-on-write fault, it checks to see whether the object taking the fault completely shadows the object below it in the chain. When it achieves complete coverage, it can bypass that object and reference the next object down the chain. The removal of its reference has the same effect as if it had exited and allowed the normal shadow chain collapse to be done with the sole remaining reference to the object. For example, in Figure 5.9, when the child of process A wrote to page 0, a copy of page 0 was made in shadow object 2. At that point object 2 completely shadows object 1, so object 2 can drop its reference to object 1 and point to the file object directly. With only one reference to object 1, the collapse of object 3 and object 1 will happen as already described. The cost of determining whether one object completely covers the object below it is much easier than tracking live references. Thus, this approach is used by FreeBSD.

This heuristic works well in practice. Even with the heavily forking parent or child processes found in many network servers, shadow-chain depth generally stabilizes at a depth of about three to four levels.

Private Snapshots

When a process makes read accesses to a private mapping of an object, it continues to see changes made to that object by other processes that are writing to the object through the filesystem or that have a shared mapping to the object. When a process makes a write access to a private mapping of an object, a snapshot of the corresponding page of the object is made and is stored in the shadow object, and the modification is made to that snapshot. Thus, further changes to that page made by other processes that are writing to the page through the filesystem or that have a shared mapping to the object are no longer visible. However, changes to unmodified pages of the object continue to be visible. This mix of changing and unchanging parts of the file can be confusing.

To provide a more consistent view of a file, a process may want to take a snapshot of the file at the time that it is initially privately mapped. Historically, both Mach and 4.4BSD provided a copy object whose effect was to take a snapshot of an object at the time that the private mapping was set up. The copy object tracked changes to an object by other processes and kept original copies of any pages that changed. None of the other UNIX vendors implemented copy objects, and there are no major applications that depend on them. The copy-object code in the virtual-memory system was large and complex, and it noticeably slowed virtual-memory performance. Consequently, copy objects were deemed unnecessary and were removed from FreeBSD as part of the early cleanup and performance work done on the virtual-memory system. Applications that want to get a snapshot of a file can do so by reading it into their address space or by making a copy of it in the filesystem and then referring to the copy.


5.6. Creation of a New Process

Processes are created with a fork system call. The fork is usually followed shortly thereafter by an exec system call that overlays the virtual address space of the child process with the contents of an executable image that resides in the filesystem. The process then executes until it terminates by exiting, either voluntarily or involuntarily, by receiving a signal. In Sections 5.6 to 5.9, we trace the management of the memory resources used at each step in this cycle.

A fork system call duplicates the address space of an existing process, creating an identical child process. The fork set of system calls is the only way that new processes are created in FreeBSD. Fork duplicates all the resources of the original process and copies that process's address space.

The virtual-memory resources of the process that must be allocated for the child include the process structure and its associated substructures, and the user structure and kernel stack. In addition, the kernel must reserve storage (either memory, filesystem space, or swap space) used to back the process. The general outline of the implementation of a fork is as follows:

  • Reserve virtual address space for the child process.

  • Allocate a process entry and thread structure for the child process, and fill it in.

  • Copy to the child the parent's process group, credentials, file descriptors, limits, and signal actions.

  • Allocate a new user structure and kernel stack, copying the current ones to initialize them.

  • Allocate a vmspace structure.

  • Duplicate the address space, by creating copies of the parent vm_map_entry structures marked copy-on-write.

  • Arrange for the child process to return 0, to distinguish its return value from the new PID that is returned to the parent process.

The allocation and initialization of the process structure, and the arrangement of the return value, were covered in Chapter 4. The remainder of this section discusses the other steps involved in duplicating a process.

Reserving Kernel Resources

The first resource to be reserved when an address space is duplicated is the required virtual address space. To avoid running out of memory resources, the kernel must ensure that it does not promise to provide more virtual memory than it is able to deliver. The total virtual memory that can be provided by the system is limited to the amount of physical memory available for paging plus the amount of swap space that is provided. A few pages are held in reserve to stage I/O between the swap area and main memory.

The reason for this restriction is to ensure that processes get synchronous notification of memory limitations. Specifically, a process should get an error back from a system call (such as sbrk, fork, or mmap) if there are insufficient resources to allocate the needed virtual memory. If the kernel promises more virtual memory than it can support, it can deadlock trying to service a page fault. Trouble arises when it has no free pages to service the fault and no available swap space to save an active page. Here, the kernel has no choice but to send a kill signal to the process unfortunate enough to be page faulting. Such asynchronous notification of insufficient memory resources is unacceptable.

Excluded from this limit are those parts of the address space that are mapped read-only, such as the program text. Any pages that are being used for a read-only part of the address space can be reclaimed for another use without being saved because their contents can be refilled from the original source. Also excluded from this limit are parts of the address space that map shared files. The kernel can reclaim any pages that are being used for a shared mapping after writing their contents back to the filesystem from which they are mapped. Here, the filesystem is being used as an extension of the swap area. Finally, any piece of memory that is used by more than one process (such as an area of anonymous memory being shared by several processes) needs to be counted only once toward the virtual-memory limit.

The limit on the amount of virtual address space that can be allocated causes problems for applications that want to allocate a large piece of address space but want to use the piece only sparsely. For example, a process may wish to make a private mapping of a large database from which it will access only a small part. Because the kernel has no way to guarantee that the access will be sparse, it takes the pessimistic view that the entire file will be modified and denies the request. One extension that many BSD-derived systems have made to the mmap system call is to add a flag that tells the kernel that the process is prepared to accept asynchronous faults in the mapping. Such a mapping would be permitted to use up to the amount of virtual memory that had not been promised to other processes. If the process then modifies more of the file than this available memory, or if the limit is reduced by other processes allocating promised memory, the kernel can then send a segmentation-fault or a more specific out-of-memory signal to the process. On receiving the signal, the process must munmap an unneeded part of the file to release resources back to the system. The process must ensure that the code, stack, and data structures needed to handle the segment-fault signal do not reside in the part of the address space that is subject to such faults.

Tracking the outstanding virtual memory accurately and determining when to limit further allocation is a complex task. Because most processes use only about half of their virtual address space, limiting outstanding virtual memory to the sum of process address spaces is needlessly conservative. However, allowing greater allocation runs the risk of running out of virtual-memory resources. Although FreeBSD calculates the outstanding-memory load, it does not enforce any total memory limit so it can be made to promise more than it can deliver. When memory resources run out, it picks a process to kill favoring processes with large memory use. An important future enhancement will be to develop a heuristic for determining when virtual-memory resources are in danger of running out and need to be limited.

Duplication of the User Address Space

The next step in fork is to allocate and initialize a new process structure. This operation must be done before the address space of the current process is duplicated because it records state in the process structure. From the time that the process structure is allocated until all the needed resources are allocated, the parent process is locked against swapping to avoid deadlock. The child is in an inconsistent state and cannot yet run or be swapped, so the parent is needed to complete the copy of its address space. To ensure that the child process is ignored by the scheduler, the kernel sets the process's state to NEW during the entire fork procedure.

Historically, the fork system call operated by copying the entire address space of the parent process. When large processes fork, copying the entire user address space is expensive. All the pages that are on secondary storage must be read back into memory to be copied. If there is not enough free memory for both complete copies of the process, this memory shortage will cause the system to begin paging to create enough memory to do the copy (see Section 5.12). The copy operation may result in parts of the parent and child processes being paged out, as well as the paging out of parts of unrelated processes.

The technique used by FreeBSD to create processes without this overhead is called copy-on-write. Rather than copy each page of a parent process, both the child and parent processes resulting from a fork are given references to the same physical pages. The page tables are changed to prevent either process from modifying a shared page. Instead, when a process attempts to modify a page, the kernel is entered with a protection fault. On discovering that the fault was caused by an attempt to modify a shared page, the kernel simply copies the page and changes the protection field for the page to allow modification once again. Only pages modified by one of the processes need to be copied. Because processes that fork typically overlay the child process with a new image with exec shortly thereafter, this technique significantly improves the performance of fork.

The next step in fork is to traverse the list of vm_map_entry structures in the parent and to create a corresponding entry in the child. Each entry must be analyzed and the appropriate action taken:

  • If the entry maps a read-only or shared region, the child can take a reference to it.

  • If the entry maps a privately mapped region (such as the data area or stack), the child must create a copy-on-write mapping of the region. The parent must be converted to a copy-on-write mapping of the region. If either process later tries to write the region, it will create a shadow map to hold the modified pages.

Map entries for a process are never merged (simplified). Only entries for the kernel map itself can be merged. The kernel-map entries need to be simplified so that excess growth is avoided. It might be worthwhile to do such a merge of the map entries for a process when it forks, especially for large or long-running processes.

With the virtual-memory resources allocated, the system sets up the kernel- and user-mode state of the new process, including the hardware memory-management registers, user structure, and stack. It then clears the NEW flag and places the process's thread on the run queue; the new process can then begin execution.

Creation of a New Process Without Copying

When a process (such as a shell) wishes to start another program, it will generally fork, do a few simple operations such as redirecting I/O descriptors and changing signal actions, and then start the new program with an exec. In the meantime, the parent shell suspends itself with wait until the new program completes. For such operations, it is not necessary for both parent and child to run simultaneously, and therefore only one copy of the address space is required. This frequently occurring set of system calls led to the implementation of the vfork system call. Although it is extremely efficient, vfork has peculiar semantics and is generally considered to be an architectural blemish.

The implementation of vfork will always be more efficient than the copy-on-write implementation because the kernel avoids copying the address space for the child. Instead, the kernel simply passes the parent's address space to the child and suspends the parent. The child process does not need to allocate any virtual-memory structures, receiving the vmspace structure and all its pieces from its parent. The child process returns from the vfork system call with the parent still suspended. The child does the usual activities in preparation for starting a new program, then calls exec. Now the address space is passed back to the parent process, rather than being abandoned, as in a normal exec. Alternatively, if the child process encounters an error and is unable to execute the new program, it will exit. Again, the address space is passed back to the parent instead of being abandoned.

With vfork, the entries describing the address space do not need to be copied, and the page-table entries do not need to be marked and then cleared of copy-on-write. Vfork is likely to remain more efficient than copy-on-write or other schemes that must duplicate the process's virtual address space. The architectural quirk of the vfork call is that the child process may modify the contents and even the size of the parent's address space while the child has control. Although modification of the parent's address space is bad programming practice, some programs have been known to take advantage of this quirk.


5.7. Execution of a File

The exec system call was described in Sections 2.4 and 3.1; it replaces the address space of a process with the contents of a new program obtained from an executable file. During an exec, the target executable image is validated and then the arguments and environment are copied from the current process image into a temporary area of pageable kernel virtual memory.

To do an exec, the system must allocate resources to hold the new contents of the virtual address space, set up the mapping for this address space to reference the new image, and release the resources being used for the existing virtual memory.

The first step is to reserve memory resources for the new executable image. The algorithm for the calculation of the amount of virtual address space that must be reserved was described in Section 5.6. For an executable that is not being debugged (and hence will not have its text space modified), a space reservation must be made for only the data and stack space of the new executable. Exec does this reservation without first releasing the currently assigned space, because the system must be able to continue running the old executable until it is sure that it will be able to run the new one. If the system released the current space and the memory reservation failed, the exec would be unable to return to the original process. Once the reservation is made, the address space and virtual-memory resources of the current process are then freed as though the process were exiting; this mechanism is described in Section 5.9.

Now the process has only a user structure and kernel stack. The kernel now allocates a new vmspace structure and creates the list of four vm_map_entry structures:

  1. A copy-on-write, fill-from-file entry maps the text segment. A copy-on-write mapping is used, rather than a read-only one, to allow active text segments to have debugging breakpoints set without affecting other users of the binary. In FreeBSD, some legacy code in the kernel debugging interface disallows the setting of break points in binaries being used by more than one process. This legacy code prevents the use of the copy-on-write feature and requires that the text be mapped read-only.

  2. A private (copy-on-write), fill-from-file entry maps the initialized data segment.

  3. An anonymous zero-fill-on-demand entry maps the uninitialized data segment.

  4. An anonymous zero-fill-on-demand entry maps the stack segment.

No further operations are needed to create a new address space during an exec system call; the remainder of the work involves copying the arguments and environment out to the top of the new stack. Initial values are set for the registers: The program counter is set to the entry point, and the stack pointer is set to point to the argument vector. The new process image is then ready to run.


5.8. Process Manipulation of Its Address Space

Once a process begins execution, it has several ways to manipulate its address space. The system has always allowed processes to expand their uninitialized data area (usually done with the malloc() library routine). The stack is grown on an as-needed basis. The FreeBSD system also allows a process to map files and devices into arbitrary parts of its address space and to change the protection of various parts of its address space, as described in Section 5.5. This section describes how these address-space manipulations are done.

Change of Process Size

A process can change its size during execution by explicitly requesting more data space with the sbrk system call. Also, the stack segment will be expanded automatically if a protection fault is encountered because of an attempt to grow the stack below the end of the stack region. In either case, the size of the process address space must be changed. The size of the request is always rounded up to a multiple of page size. New pages are marked fill-with-zeros, since there are no contents initially associated with new sections of the address space.

The first step of enlarging a process's size is to check whether the new size would violate the size limit for the process segment involved. If the new size is in range, the following steps are taken to enlarge the data area:

1. Verify that the virtual-memory resources are available.

2. Verify that the address space of the requested size immediately following the current end of the data area is not already mapped.

3. If the existing vm_map_entry is the only reference to the swap object, increment the vm_map_entry's ending address by the requested size and increase the size of the swap object by the same amount. If the swap object has two or more references (as it would after a process forked), a new vm_map_entry must be created with a starting address immediately following the end of the previous fixed-sized entry. Its ending address is calculated to give it the size of the request. It will be backed by a new swap object. Until this process forks again, the new entry and its swap object will be able to continue growing.

If the change is to reduce the size of the data segment, the operation is easy: Any memory allocated to the pages that will no longer be part of the address space is freed. The ending address of the vm_map_entry is reduced by the size. If the requested size reduction is bigger than the range defined by the vm_map_entry, the entire entry is freed, and the remaining reduction is applied to the vm_map_entry that precedes it. This algorithm is applied until the entire reduction has been made. Future references to these addresses will result in protection faults, as access is disallowed when the address range has been deallocated.

File Mapping

The mmap system call requests that a file be mapped into an address space. The system call may request either that the mapping be done at a particular address or that the kernel pick an unused area. If the request is for a particular address range, the kernel first checks to see whether that part of the address space is already in use. If it is in use, the kernel first does an munmap of the existing mapping, then proceeds with the new mapping.

The kernel implements the mmap system call by traversing the list of vm_map_entry structures for the process. The various overlap conditions to consider are shown in Figure 5.10. The five cases are as follows:

  1. The new mapping exactly overlaps an existing mapping. The old mapping is deallocated as described in Section 5.9. The new mapping is created in its place as described in the paragraph following this list.

  2. The new mapping is a subset of the existing mapping. The existing mapping is split into three pieces (two pieces if the new mapping begins at the beginning or ends at the end of the existing mapping). The existing vm_map_entry structure is augmented with one or two additional vm_map_entry structures: one mapping the remaining part of the existing mapping before the new mapping, and one mapping the remaining part of the existing mapping following the new mapping. Its overlapped piece is replaced by the new mapping, as described in the paragraph following this list.

  3. The new mapping is a superset of an existing mapping. The old mapping is deallocated as described in Section 5.9, and a new mapping is created as described in the paragraph following this list.

  4. The new mapping starts partway into and extends past the end of an existing mapping. The existing mapping has its length reduced by the size of the unmapped area. Its overlapped piece is replaced by the new mapping, as described in the paragraph following this list.

  5. The new mapping extends into the beginning of an existing mapping. The existing mapping has its starting address incremented and its length reduced by the size of the covered area. Its overlapped piece is replaced by the new mapping, as described in the paragraph following this list.

Figure 5.10. Five types of overlap that the kernel must consider when adding a new address mapping.

[View full size image]

In addition to these five basic types of overlap, a new mapping request may span several existing mappings. Specifically, a new request may be composed of zero or one of type 4, mzero to many of type 3, and zero or one of type 5. When a mapping is shortened, any shadow pages associated with it are released, because they are no longer needed.

Once the address space is zero filled, the kernel creates a new vm_map_entry to describe the new address range. If the object being mapped is already being mapped by another process, the new entry gets a reference to the existing object. This reference is obtained in the same way, as described in Section 5.6, when a new process is being created and needs to map each of the regions in its parent. If this request is a mapping of a file, then the kernel sets the new vm_map_entry to reference its object.

If this is a mapping to an anonymous region, then a new swap object must be created. First, a new object is allocated with a pager type of swap (pagers are described in Section 5.10). The vm_map_entry is then set to reference the object.

Change of Protection

A process may change the protections associated with a region of its virtual memory by using the mprotect system call. The size of the region to be protected may be as small as a single page. Because the kernel depends on the hardware to enforce the access permissions, the granularity of the protection is limited by the underlying hardware. A region may be set for any combination of read, write, and execute permissions. Many architectures do not distinguish between read and execute permissions; on such architectures, the execute permission is treated as read permission.

The kernel implements the mprotect system call by finding the existing vm_map_entry structure or structures that cover the region specified by the call. If the existing permissions are the same as the request, then no further action is required. Otherwise, the new permissions are compared to the maximum protection value associated with the vm_map_entry. The maximum value is set at mmap time and reflects the maximum value allowed by the underlying file. If the new permissions are valid, one or more new vm_map_entry structures have to be set up to describe the new protections. The set of overlap conditions that must be handled is similar to that described in the previous subsection. Instead of replacing the object underlying the new vm_map_entry structures, these vm_map_entry structures still reference the same object; the difference is that they grant different access permissions to it.


5.9. Termination of a Process

The final change in process state that relates to the operation of the virtual-memory system is exit; this system call terminates a process, as described in Chapter 4. The part of exit that is discussed here is the release of the virtual-memory resources of the process. There are two sets of virtual-memory resources that need to be freed:

  1. The user portions of the address space, both memory and swap space

  2. The user structure and kernel stack

The first set of resources is freed in exit. The second set of resources is freed in wait. The release of the second set of resources is delayed because the kernel stack must be used until the process relinquishes the processor for the final time.

The first step—freeing the user address space—is identical to the one that occurs during exec to free the old address space. The free operation proceeds entry by entry through the list of vm_map_entry structures associated with the address space. The first step in freeing an entry is to traverse its list of shadow objects. If the entry is the last reference to a shadow object, then any memory or swap space that is associated with the object can be freed. In addition, the machine-dependent routines are called to unmap and free up any page table or data structures that are associated with the object. If the shadow object is still referenced by other vm_map_entry structures, its resources cannot be freed, but the kernel still needs to call the machine-dependent routines to unmap and free the resources associated with the current process mapping. Finally, if the underlying object referenced by the vm_map_entry is losing its last reference, then that object is a candidate for deallocation. If it is an object that will never have any chance of a future reuse (such as an anonymous object associated with a stack or uninitialized data area), then its resources are freed as though it were a shadow object. However, if the object is associated with a vnode (e.g., it maps a file such as an executable), the object will persist until the vnode is reused for another purpose. Until the vnode is reused, the object and its associated pages will be available for reuse by newly executing processes or by processes mapping in a file.

With all its resources free, the exiting process finishes detaching itself from its process group and notifies its parent that it is done. The process has now become a zombie process—one with no resources. Its parent will collect its exit status with a wait call. Because the process structure, user structure, and kernel stack are allocated using the zone allocator, they will normally be retained for future use by another process rather than being broken down and their memory pages reclaimed. Thus, there is nothing for the virtual-memory system to do when wait is called: All virtual-memory resources of a process are removed when exit is done. On wait, the system just returns the process status to the caller, releases the process structure, user structure, and kernel stack back to the zone allocator, and frees the space in which the resource-usage information was kept.


5.10. The Pager Interface

The pager interface provides the mechanism by which data are moved between backing store and physical memory. The FreeBSD pager interface is an evolution of the interface present in Mach 2.0 as evolved by 4.4BSD. The interface is page based, with all data requests made in multiples of the software page size. Vm_page structures are passed around as descriptors providing the backing-store offset and physical-memory address of the desired data. This interface should not be confused with the Mach 3.0 external paging interface [Young, 1989], where pagers are typically user applications outside the kernel and are invoked via asynchronous remote procedure calls using the Mach interprocess-communication mechanism. The FreeBSD interface is internal in the sense that the pagers are compiled into the kernel and pager routines are invoked via simple function calls.

Each virtual-memory object has a pager type, pager handle, and pager private data associated with it. Conceptually, the pager describes a logically contiguous piece of backing store, such as a chunk of swap space or a disk file. The pager type identifies the pager responsible for supplying the contents of pages within the object. Each pager registers a set of functions that define its operations. These function sets are stored in an array indexed by pager type. When the kernel needs to do a pager operation, it uses the pager type to index into the array of pager functions and then selects the routine that it needs such as getting or putting pages. For example,

(*pagertab[object->type]->pgo_putpages)
    (object, vmpage, count, flags, rtvals);

pushes count pages starting with page vmpage from object.

A pager type is specified when an object is created to map a file, device, or piece of anonymous memory into a process address space. The pager manages the object throughout its lifetime. When a page fault occurs for a virtual address mapping a particular object, the fault-handling code allocates a vm_page structure and converts the faulting address to an offset within the object. This offset is recorded in the vm_page structure, and the page is added to the list of pages cached by the object. The page frame and object are then passed to the underlying pager routine. The pager routine is responsible for filling the vm_page structure with the appropriate initial value for that offset of the object that it represents.

The pager is also responsible for saving the contents of a dirty page if the system decides to push out the latter to backing store. When the pageout daemon decides that a particular page is no longer needed, it requests the object that owns the page to free the page. The object first passes the page with the associated logical offset to the underlying pager to be saved for future use. The pager is responsible for finding an appropriate place to save the page, doing any I/O necessary for the save, and then notifying the object that the page is being freed. When it is done, the pager marks the page as clean so that the pageout daemon can move the vm_page structure to the cache or free list for future use.

There are seven routines associated with each pager type; see Table 5.1. The pgo_init() routine is called at boot time to do any one-time type-specific initializations, such as allocating a pool of private pager structures. The pgo_alloc() routine associates a pager with an object as part of the creation of the object. The pgo_dealloc() routine disassociates a pager from an object as part of the destruction of the object. Objects are created either by the mmap system call or internally as part of the creation of a process as part of the fork or exec system calls.

Table 5.1. Operations defined by a pager.

Operation

Description

Pgo_init()

initialize pager

pgo_alloc()

allocate pager

pgo_dealloc()

deallocate pager

pgo_getpages()

read page(s) from backing store

pgo_putpages()

write page(s) to backing store

pgo_haspage()

check whether backing store has a page

pgo_pageunswapped()

remove a page from backing store (swap pager only)

Pgo_getpages() is called to return one or more pages of data from a pager. The main use of this routine is by the page-fault handler. Pgo_putpages() writes back one or more pages of data. This routine is called by the pageout daemon to write back one or more pages asynchronously, and by msync to write back one or more pages synchronously or asynchronously. Both the get and put routines are called with an array of vm_page structures and a count indicating the affected pages.

The pgo_haspage() routine queries a pager to see whether it has data at a particular backing-store offset. This routine is used in the clustering code of the page-fault handler to determine whether pages on either side of a faulted page can be read in as part of a single I/O operation. It is also used when collapsing objects to determine if the allocated pages of a shadow object completely obscure the allocated pages of the object that it shadows.

The four types of pagers supported by the system are described in the next four subsections.

Vnode Pager

The vnode pager handles objects that map files in a filesystem. Whenever a file is opened either explicitly by open or implicitly by exec, the system must find an existing vnode that represents it or, if there is no existing vnode for the file, allocate a new vnode for it. Part of allocating a new vnode is to allocate an object to hold the pages of the file and to associate the vnode pager with the object. The object handle is set to point to the vnode and the private data stores the size of the file. Any time the vnode changes size, the object is informed by a call to vnode_pager_setsize().

When a pagein request is received by the vnode-pager pgo_getpages() routine, it is passed an array of physical pages, the size of the array, and the index into the array of the page that is required. Only the required page must be read, but the pgo_getpages() routine is encouraged to provide as many of the others as it can easily read at the same time. For example, if the required page is in the middle of the block of a file, the filesystem will usually read the entire file block since the file block can be read with a single I/O operation. The larger read will fill in the required page along with the pages surrounding it. The I/O is done using a physical-I/O buffer that maps the pages to be read into the kernel address space long enough for the pager to call the device-driver strategy routine to load the pages with the file contents. Once the pages are filled, the kernel mapping can be dropped, the physical-I/O buffer can be released, and the pages can be returned.

When the vnode pager is asked to save a page to be freed, it simply arranges to write the page back to the part of the file from which the page came. The request is made with the pgo_getpages() routine, which is passed an array of physical pages, the size of the array, and the index into the array of the page that must be written. Only the required page must be written, but the pgo_putpages() routine is encouraged to write as many of the others as it can easily handle at the same time. The filesystem will write out all the pages that are in the same filesystem block as the required page. As with the pgo_getpages() routine, the pages are mapped into the kernel only long enough to do the write operation.

If a file is being privately mapped, then modified pages cannot be written back to the filesystem. Such private mapping must use a shadow object with a swap pager for all pages that are modified. Thus, a privately mapped object will never be asked to save any dirty pages to the underlying file.

Historically, the BSD kernel had separate caches for the filesystems and the virtual memory. FreeBSD has eliminated the filesystem buffer cache by replacing it with the virtual-memory cache. Each vnode has an object associated with it, and the blocks of the file are stored in the pages associated with the object. The file data is accessed using the same pages whether they are mapped into an address space or accessed via read and write. An added benefit of this design is that the filesystem cache is no longer limited by the address space in the kernel that can be dedicated to it. Absent other demands on the system memory, it can all be dedicated to caching filesystem data.

Device Pager

The device pager handles objects representing memory-mapped hardware devices. Memory-mapped devices provide an interface that looks like a piece of memory. An example of a memory-mapped device is a frame buffer, which presents a range of memory addresses with one word per pixel on the screen. The kernel provides access to memory-mapped devices by mapping the device memory into a process's address space. The process can then access that memory without further operating-system intervention. Writing to a word of the frame-buffer memory causes the corresponding pixel to take on the appropriate color and brightness.

The device pager is fundamentally different from the other three pagers in that it does not fill provided physical-memory pages with data. Instead, it creates and manages its own vm_page structures, each of which describes a page of the device space. The head of the list of these pages is kept in the pager private-data area of the object. This approach makes device memory look like wired physical memory. Thus, no special code should be needed in the remainder of the virtual-memory system to handle device memory.

When a device is first mapped, the device-pager allocation routine will validate the desired range by calling the device d_mmap () routine. If the device allows the requested access for all pages in the range, an empty page list is created in the private-data area of the object that manages the device mapping. The device-pager allocation routine does not create vm_page structures immediately— they are created individually by the pgo_getpages() routine as they are referenced. The reason for this late allocation is that some devices export a large memory range in which either not all pages are valid or the pages may not be accessed for common operations. Complete allocation of vm_page structures for these sparsely accessed devices would be wasteful.

The first access to a device page will cause a page fault and will invoke the device-pager pgo_getpages() routine. The device pager creates a vm_page structure, initializes the latter with the appropriate object offset and a physical address returned by the device d_mmap() routine, and flags the page as fictitious. This vm_page structure is added to the list of all such allocated pages for the object. Since the fault code has no special knowledge of the device pager, it has preallocated a physical-memory page to fill and has associated that vm_page structure with the object. The device-pager routine removes that vm_page structure from the object, returns the structure to the free list, and inserts its own vm_page structure in the same place.

The device-pager pgo_putpages() routine expects never to be called and will panic if it is. This behavior is based on the assumption that device-pager pages are never entered into any of the paging queues and hence will never be seen by the pageout daemon. However, it is possible to msync a range of device memory. This operation brings up an exception to the higher-level virtual-memory system's ignorance of device memory: The object page-cleaning routine will skip pages that are flagged as fictitious.

Finally, when a device is unmapped, the device-pager deallocation routine is invoked. This routine deallocates all the vm_page structures that it allocated.

Physical-Memory Pager

The physical-memory pager handles objects that contain nonpagable memory. Their only use is for the System V shared-memory interface, which can be configured to use nonpagable memory instead of the default swappable memory.

The first access to a physical-memory-pager page will cause a page fault and will invoke the pgo_getpages() routine. Like the swap pager, the physical-memory pager zero-fills pages when they are first faulted. Unlike the swap pager, the page is marked as unmanaged so that it will not be considered for replacement by the page-out daemon. Marking its pages unmanaged makes the memory for the physical-memory pager look like wired physical memory. Thus, no special code is needed in the remainder of the virtual-memory system to handle physical-memory-pager memory.

The pgo_putpages() routine of the physical-memory-pager does not expect to be called, and it panics if it is. This behavior is based on the assumption that physical-memory-pager pages are never entered into any of the paging queues and hence will never be seen by the pageout daemon. However, it is possible to msync a range of device memory. This operation brings up an exception to the higher-level virtual-memory system's ignorance of physical-memory-pager memory: The object page-cleaning routine will skip pages that are flagged as unmanaged.

Finally, when an object using a physical-memory-pager is freed, all its pages have their unmanaged flag cleared and are released back to the list of free pages.

Swap Pager

The term swap pager refers to two functionally different pagers. In the most common use, swap pager refers to the pager that is used by objects that map anonymous memory. This pager has sometimes been referred to as the default pager because it is the pager that is used if no other pager has been requested. It provides what is commonly known as swap space: nonpersistent backing store that is zero filled on first reference. When an anonymous object is first created, it is assigned the default pager. The default pager allocates no resources and provides no storage backing. The default pager handles page faults (pgo_getpage()) by zero filling and page queries (pgo_haspage()) as not held. The expectation is that free memory will be plentiful enough that it will not be necessary to swap out any pages. The object will simply create zero-filled pages during the process lifetime that can all be returned to the free list when the process exits. When an object is freed with the default pager, no pager cleanup is required, since no pager resources were allocated.

However, on the first request by the pageout daemon to remove an active page from an anonymous object, the default pager replaces itself with the swap pager. The role of the swap pager is swap-space management: figuring out where to store dirty pages and how to find dirty pages when they are needed again. Shadow objects require that these operations be efficient. A typical shadow object is sparsely populated: It may cover a large range of pages, but only those pages that have been modified will be in the shadow object's backing store. In addition, long chains of shadow objects may require numerous pager queries to locate the correct copy of an object page to satisfy a page fault. Hence, determining whether a pager contains a particular page needs to be fast, preferably requiring no I/O operations. A final requirement of the swap pager is that it can do asynchronous writeback of dirty pages. This requirement is necessitated by the pageout daemon, which is a single-threaded process. If the pageout daemon blocked waiting for a page-clean operation to complete before starting the next operation, it is unlikely that it could keep enough memory free in times of heavy memory demand.

In theory, any pager that meets these criteria can be used as the swap pager. In Mach 2.0, the vnode pager was used as the swap pager. Special paging files could be created in any filesystem and registered with the kernel. The swap pager would then suballocate pieces of the files to back particular anonymous objects. One obvious advantage of using the vnode pager is that swap space can be expanded by the addition of more swap files or the extension of existing ones dynamically (i.e., without rebooting or reconfiguring of the kernel). The main disadvantage is that the filesystem does not provide as much bandwidth as direct access to the disk.

The desire to provide the highest possible disk bandwidth led to the creation of a special raw-partition pager to use as the swap pager for FreeBSD. Previous versions of BSD also used dedicated disk partitions, commonly known as swap partitions, so this partition pager became the swap pager. The remainder of this section describes how the swap pager is implemented and how it provides the necessary capabilities for backing anonymous objects.

In 4.4BSD, the swap pager preallocated a fixed-sized structure to describe the object. For a large object the structure would be large even if only a few pages of the object were pushed to backing store. Worse, the size of the object was frozen at the time of allocation. Thus, if the anonymous area continued to grow (such as the stack or heap of a process), a new object had to be created to describe the expanded area. On a system that was short of memory, the result was that a large process could acquire many anonymous objects. Changing the swap pager to handle growing objects dramatically reduced this object proliferation. Other problems with the 4.4BSD swap pager were that its simplistic management of the swap space led to fragmentation, slow allocation under load, and deadlocks brought on by its need to allocate kernel memory during periods of shortage. For all these reasons, the swap pager was completely rewritten in FreeBSD 4.0.

Swap space tends to be sparsely allocated. On average, a process only accesses about half of its allocated address space during its lifetime. Thus, only about half the pages in an object ever come into existence. Unless the machine is under heavy memory pressure and the process is long-lived, most of the pages in the object that do come into existence will never be pushed to backing store. So the new swap pager replaced the old fixed-size block map for each object with a method that allocates a structure for each set of swap blocks that gets allocated. Each structure can track a set of up to 32 contiguous swap blocks. A large object with two pages swapped out will use at most two of these structures, and only one if the two swapped pages are close to each other (as they often are). The amount of memory required to track swap space for an object is proportional to the number of pages that have been pushed to swap rather than to the size of the object. The size of the object is no longer frozen when its first page is swapped out, since any pages that are part of its larger size can be accommodated.

The structures that track swap space usage are kept in a global hash table managed by the swap pager. While it might seem logical to store the structures separately on lists associated with the object of which they are a part, the single global hash table has two important advantages:

  1. It ensures a short time to determine whether a page of an object has been pushed to swap. If the structures were linked onto a list headed by the object, then objects with many swapped pages would require the traversal of a long list. The long list could be shortened by creating a hash table for every object, but that would require much more memory than simply allocating a single large hash table that could be used by all objects.

  2. It allows operations that need to scan all the allocated swap blocks to have a centralized place to find them rather than needing to scan all the anonymous objects in the system. An example is the swapoff system call that removes a swap partition from use. It needs to page in all the blocks from the device that is to be taken out of service.

The free space in the swap area is managed with a bitmap with one bit for each page-sized block of swap space. The bitmap for the entire swap area is allocated when the swap space is first added to the system. This initial allocation avoids the need to allocate kernel memory during critical low-memory swapping operations. Although when it is first allocated the bitmap requires more space than the block list needed when first allocated, its size does not change with use, whereas the block list grew to a size bigger than the bitmap as the swap area became fragmented. The system tends to swap when it is low on memory. To avoid potential deadlocks, kernel memory should not be allocated at such times.

Doing a linear scan of the swap-block bitmaps to find free space would be unacceptably slow. Thus, the bitmap is organized in a radix-tree structure with free-space hinting in the radix-node structures. The use of radix-tree structures makes swap-space allocation and release a constant-time operation. To reduce fragmentation, the radix tree can allocate large contiguous chunks at once, skipping over smaller fragmented chunks.

A future improvement would be to keep track of the different-sized free areas as swap allocations are done similarly to the way that the filesystem tracks the different sizes of free space. This free-space information would increase the probability of doing contiguous allocation and improve locality of reference.

Swap blocks are allocated at the time that swap out is done. They are freed when the page is brought back in and becomes dirty or the object is freed.

The swap pager is responsible for managing the I/O associated with the pgo_putpages() request. Once it identifies the set of pages within the pgo_putpages() request that it will be able to write, it must allocate a buffer and have those pages mapped into it. Because the swap pager does not synchronously wait while the I/O is done, it does not regain control after the I/O operation completes. Therefore, it marks the buffer with a callback flag and sets the routine for the callback to be swp_pager_async_iodone().

When the push completes, swp_pager_async_iodone() is called. Each written page is marked as clean, has its busy bit cleared, and calls the vm_page_io_finish() routine to notify the pageout daemon that the write has completed and to awaken any processes waiting for it. The swap pager then unmaps the pages from the buffer and releases it. A count of pageouts-in-progress is kept for the pager associated with each object; this count is decremented when the pageout completes, and, if the count goes to zero, a wakeup() is issued. This operation is done so that an object that is deallocating a swap pager can wait for the completion of all page-out operations before freeing the pager's references to the associated swap space.

Because the number of swap buffers is constant, the swap pager must take care to ensure that it does not use more than its fair share. Once this limit is reached, the pgo_putpages() operations blocks until one of the swap pager's outstanding writes completes. This unexpected blocking of the pageout daemon is an unfortunate side effect of pushing the buffer management down into the pagers. Any single pager hitting its buffer limit stops the pageout daemon. While the page-out daemon might want to do additional I/O operations using other I/O resources such as the network, it is prevented from doing so. Worse, the failure of any single pager can deadlock the system by preventing the pageout daemon from running.


5.11. Paging

When the memory-management hardware detects an invalid virtual address, it generates a trap to the system. This page-fault trap can occur for several reasons. Most BSD programs are created in a format that permits the executable image to be paged into main memory directly from the filesystem. When a program in a demand-paged format is first run, the kernel marks as invalid the pages for the text and initialized-data regions of the executing process. The text and initialized data regions share an object that provides fill-on-demand from the filesystem. As part of mapping in the object, the kernel traverses the list of pages associated with the object and marks them as resident and copy-on-write in the newly created process. For a heavily used executable with most of its pages already resident, this prepaging reduces many of its initial page faults. As missing pages of the text or initialized-data region are first referenced, or write attempts are made on pages in the initialized-data region, page faults occur.

Page faults can also occur when a process first references a page in the uninitialized-data region of a program. Here, the anonymous object managing the region automatically allocates memory to the process and initializes the newly assigned page to zero. The idle process fills part of its time by zeroing out pages on the free list so that the system does not have to do the zero fill when servicing the first reference to a page for an anonymous object. Other types of page faults arise when previously resident pages have been reclaimed by the system in response to a memory shortage.

The handling of page faults is done with the vm_fault() routine; this routine services all page faults. Each time vm_fault() is invoked, it is provided the virtual address that caused the fault. The first action of vm_fault() is to traverse the vm_map_entry list of the faulting process to find the entry associated with the fault. The routine then computes the logical page within the underlying object and traverses the list of objects to find or create the needed page. Once the page has been found, vm_fault() must call the machine-dependent layer to validate the faulted page and return to restart the process.

The details of calculating the address within the object are described in Section 5.4. Having computed the offset within the object and determined the object's protection and object list from the vm_map_entry, the kernel is ready to find or create the associated page. The page-fault-handling algorithm is shown in Figure 5.11. In the following overview, the lettered points are references to the tags down the left side of the code.

Figure 5.11. Page-fault handling.
/*
 * Handle a page fault occurring at the given address,
 * requiring the given permissions, in the map specified.
 * If successful, insert the page into the associated
 * physical map.
 */
int vm_fault(
    vm_map_t map,
    vm_offset_t addr,
    vm_prot_t type)
{
RetryFault:
    lookup address in map returning object/offset/prot;
    first_object = object;
[A] for (;;) {
        page = lookup page at object/offset;
[B]     if (page found) {
            if (page busy)
                block and goto RetryFault;
            remove from paging queues;
            mark page as busy;
            break;
        }
[C]     if (object has nondefault pager or
            object == first_object) {
            page = allocate a page for object/offset;
            if (no pages available)
                block and goto RetryFault;
        }
[D]     if (object has nondefault pager) {
            scan for pages to cluster;
            call pager to fill page(s);
            if (IO error)
                return an error;
            if (pager has page)
                break;
            if (object != first_object)
                free page;
        }
        /* no pager, or pager does not have page */
[E]     if (object == first_object)
            first_page = page;
        next_object = next object;
[F]     if (no next object) {
            if (object != first_object) {
                object = first_object;
                page = first_page;
            }
            first_page = NULL;
            zero fill page;
            break;
        }
        object = next_object;
    }
[G] /* appropriate page has been found or allocated */
    orig_page = page;
[H] if (object != first object) {
        if (fault type ==WRITE) {
            copy page to first_page;
            deactivate page;
            page =first_page;
            object =first_object;
        } else {
            prot &=~WRITE;
            mark page copy-on-write;
        }
    }
[I] if (prot &WRITE)
        mark page not copy-on-write;
    enter mapping for page;
    enter read-only mapping for clustered pages;
[J] activate and unbusy page;
    if (first_page != NULL)
        unbusy and free first_page;
}

  1. The loop traverses the list of shadow, anonymous, and file objects until it either finds an object that holds the sought-after page or reaches the final object in the list. If no page is found, the final object will be requested to produce it.

  2. An object with the desired page has been found. If the page is busy, another process may be in the middle of faulting it in, so this process is blocked until the page is no longer busy. Since many things could have happened to the affected object while the process was blocked, it must restart the entire fault-handling algorithm. If the page was not busy, the algorithm exits the loop with the page.

  3. Anonymous objects (such as those used to represent shadow objects) do not upgrade from the default pager to the swap pager until the first time that they need to push a page to backing store. Thus, if an object has a pager other than the default pager, then there is a chance that the page previously existed but was paged out. If the object has a nondefault pager, then the kernel needs to allocate a page to give to the pager to be filled (see D). The special case for the object being the first object is to avoid a race condition with two processes trying to get the same page. The first process through will create the sought-after page in the first object but keep it marked as busy. When the second process tries to fault the same page, it will find the page created by the first process and block on it (see B). When the first process completes the pagein processing, it will unlock the first page, causing the second process to awaken, retry the fault, and find the page created by the first process.

  4. Before calling the pager, check to see if any of the eight pages on either side of the faulting page are eligible to be brought in at the same time. To be eligible, a page must be part of the object and neither already in memory nor part of another I/O operation. The pager is given the range of possible pages and told which one is the required page. It must return the required page if it holds a copy of it. The other pages are produced only if they are held by the object and can be easily read at the same time. If the required page is present in the file or swap area, the pager will bring it back into the newly allocated page. If the pagein succeeds, then the sought-after page has been found. If the page never existed, then the pagein request will fail. Unless this object is the first, the page is freed and the search continues. If this object is the first, the page is not freed, so it will act as a block to further searches by other processes (as described in C).

  5. If the kernel created a page in the first object but did not use that page, it will have to remember that page so it can free the page when the pagein is done (see J).

  6. If the search has reached the end of the object list and has not found the page, then the fault is on an anonymous object chain, and the first object in the list will handle the page fault using the page allocated in C. The first_page entry is set to NULL to show that it does not need to be freed, if the idle process has not already done so the page is zero filled, and the loop is exited.

  7. The search exits the loop with page as the page that has been found or allocated and initialized, and object as the owner of that page. The page has been filled with the correct data at this point.

  8. If the object providing the page is not the first object, then this mapping must be private, with the first object being a shadow object of the object providing the page. If pagein is handling a write fault, then the contents of the page that it has found have to be copied to the page that it allocated for the first object. Having made the copy, it can release the object and page from which the copy came, since the first object and first page will be used to finish the page-fault service. If pagein is handling a read fault, it can use the page that it found, but it has to mark the page copy-on-write to avoid the page being modified in the future.

  9. If pagein is handling a write fault, then it has made any copies that were necessary, so it can safely make the page writable. As any pages around the required page that were brought into memory as part of the clustering were not copied, they are mapped read-only so that if a write is done on one of them, the full page-fault analysis will be done and a copy made at that time if it is necessary to do so.

  10. As the page and possibly the first_page are released, any processes waiting for that page of the object will get a chance to run to get their own references.

Note that the page and object locking has been elided in Figure 5.11 to simplify the explanation.

Hardware-Cache Design

Because the speed of CPUs has increased far more rapidly than the speed of main memory, most machines today require the use of a memory cache to allow the CPU to operate near its full potential.

Code that describes the operation of a hardware cache is given in Figure 5.12. An actual cache is entirely implemented in hardware, so the loop shown in Figure 5.12 would really be done by parallel comparisons rather than iteratively. Historically most machines had a direct-mapped cache. With a direct-mapped cache, an access to byte N followed by an access to byte N + (CACHELINES x LINESIZE) would cause the cached data for byte N to be lost. Most modern caches are either 2-way set associative or 4-way set associative. The 4-way set associative cache allows access of four different memory regions that overlap the same cache memory without destroying the previously cached data. But on the fifth access at that offset, an earlier cached value is lost.

There are several cache-design choices that require cooperation with the virtual-memory system. The design option with the biggest effect is whether the cache uses virtual or physical addressing. A physically addressed cache takes the address from the CPU, runs it through the MMU to get the address of the physical page, then uses this physical address to find out whether the requested memory location is available in the cache. Although a translation lookaside buffer (described in Section 5.13) significantly reduces the average latency of the translation, there is still a delay in going through the MMU. A virtually addressed cache uses the virtual address as that address comes from the CPU to find out whether the requested memory location is available in the cache. The virtual-address cache is faster than the physical-address cache because it avoids the time to run the address through the MMU. However, the virtual-address cache must be flushed completely after each context switch, because virtual addresses from one process are indistinguishable from the virtual addresses of another process. By contrast, a physical-address cache needs to flush only a few individual entries when their associated physical page is reassigned. In a system with many short-running processes, a virtual-address cache gets flushed so frequently that it is seldom useful.

Figure 5.12. Hardware-cache algorithm. Key: LINESIZE—Number of bytes in each cache line, typically 64 or 128 bytes; CACHELINES—Number of lines in the cache, 8192 is a typical size; SETSIZE—1 for a direct mapped cache, 2 for 2-way set associative, or 4 for 4-way set associative.
struct cache {
    vm_offset_t key;             /* address of data */
    char        data[LINESIZE];  /* cache data */
} cache[CACHELINES][SETSIZE];

/*
 * If present, get data for addr from cache. Otherwise fetch
 * data from main memory, place in cache, and return it.
 */
hardware_cache_fetch(vm_offset_t addr)
{
    vm_offset_t key, line;

    line = (addr / LINESIZE) % CACHELINES;
    for (key = 0; key < SETSIZE; key++)
        if (cache[line][key] .key == addr)
            break;
    if (key < SETSIZE)
        return (cache[line][key].data);
    key = select_replacement_key(line);
    cache[line][key].key = addr;
    return (cache[line][key].data = fetch_from_RAM(addr));
}

A further refinement to the virtual-address cache is to add a process tag to the key field for each cache line. At each context switch, the kernel loads a hardware context register with the tag assigned to the process. Each time an entry is recorded in the cache, both the virtual address and the process tag that faulted it are recorded in the key field of the cache line. The cache looks up the virtual address as before, but when it finds an entry, it compares the tag associated with that entry to the hardware context register. If they match, the cached value is returned. If they do not match, the correct value and current process tag replace the old cached value. When this technique is used, the cache does not need to be flushed completely at each context switch, since multiple processes can have entries in the cache. The drawback is that the kernel must manage the process tags. Usually, there are fewer tags (8 to 16) than there are processes. The kernel must assign the tags to the active set of processes. When an old process drops out of the active set to allow a new one to enter, the kernel must flush the cache entries associated with the tag that it is about to reuse.

A final consideration is a write-through versus a write-back cache. A write-through cache writes the data back to main memory at the same time as it is writing to the cache, forcing the CPU to wait for the memory access to conclude.

A write-back cache writes the data to only the cache, delaying the memory write until an explicit request or until the cache entry is reused. The write-back cache allows the CPU to resume execution more quickly and permits multiple writes to the same cache block to be consolidated into a single memory write. However, the writes must be forced any time it is necessary for the data to be visible to other CPUs on a multiprocessor.

Page Coloring

Page coloring is a performance optimization designed to ensure that accesses to contiguous pages in virtual memory make the best use of a physical-address cache. The Intel architecture uses a physical-address cache referred to as the L1 cache. The performance of the system is closely tied to the percentage of memory references that can be serviced by the L1 cache because the L1 cache operates at the processor frequency. A miss in the L1 cache requires a few cycles of processor delay to wait for data in the L2 cache, or tens of cycles to wait for the dynamic RAM that stores the main memory.

A process on FreeBSD is running with a virtual-address space, not a physical-address space. The physical pages underlying the virtual-address space are rarely contiguous. In the worst case, two consecutive virtual pages could be stored in two physical pages backed by the same location in the L1 cache. Consider a process running on a CPU with a 1 Mbyte physical cache. If the process has its first virtual page mapped at physical page 1 Mbyte and its second virtual page mapped at physical page 2 Mbyte, despite maintaining locality of reference between its two consecutive virtual pages, the process unintentionally keeps missing the L1 cache. The effect is mitigated somewhat with multiway set-associative caches, but it is still problematic.

The role of the page-coloring algorithm is to ensure that consecutive pages in virtual memory will be contiguous from the point of view of the cache. Instead of assigning random physical pages to virtual addresses, page coloring assigns cache-contiguous physical pages to virtual addresses. Two physical pages are cache-contiguous if they hit consecutive pages in the cache. These pages may be far apart in physical memory. Figure 5.13 shows a sample assignment of pages to a process. In Figure 5.13, the cache holds 4 pages and page 10 of physical memory is assigned to page 0 of the process's virtual memory. Physical-memory page 10 is color 2 of the cache (10 modulo 4 equals 2). The page coloring code has assigned page 15 of physical memory to page 1 of the process's virtual memory as it has color 3. The page-coloring code attempts to avoid assigning pages 6 or 14 because they map over the same cache memory as page 10 and would result in nonoptimal caching, whereas any of pages 7, 11, or 15 are optimal, since they all follow the cache memory used for page 10.

Figure 5.13. Assignment of physical pages to a process.

[View full size image]

Page coloring is implemented by creating as many free lists as there are pages in the cache. Each of the pages in the cache is assigned a color. A machine with 1 Mbyte of cache and 4 Kbyte pages would have 256 cache pages—hence 256 colors. At system startup, each vm_page records its color in the L1 cache. When a page is freed, it is put onto the free list corresponding to its color. When an object is created, it is assigned a starting color. To ensure that demand for colors is moderately balanced, the starting color to be used for the next object is updated by the smaller of the size of the object or a third of the size of the L1 cache. When a page fault occurs on an object, its preferred color for the new page is calculated by adding the faulting logical-page number of the object to the starting color of the object modulo the number of colors. A page is taken from the free list of that color if it is available. Otherwise colors furthest away from the requested color are checked until an available page is found. A page furthest away is used because the algorithm wants to minimize the possibility of having a cache collision with a neighboring page in the virtual address space. For example, if a next-higher-colored page were used, it would likely collide with the page that the algorithm would select for the next-higher virtual address.

Page coloring makes virtual memory as deterministic as physical memory from the perspective of cache performance. Thus, programs can be written under the assumption that the characteristics of the underlying hardware cache are the same for their virtual address space as they would be if the program had been run directly in a physical-address space.


5.12. Page Replacement

The service of page faults and other demands for memory may be satisfied from the free list for some time, but eventually memory must be reclaimed for reuse. Some pages are reclaimed when processes exit. On systems with a large amount of memory and low memory demand, exiting processes may provide enough free memory to fill demand. This case arises when there is enough memory for the kernel and for all pages that have ever been used by any current process. Obviously, many computers do not have enough main memory to retain all pages in memory. Thus, it eventually becomes necessary to move some pages to secondary storage—back to the filesystem and to the swap space. Bringing in a page is demand driven. For paging it out, however, there is no immediate indication when a page is no longer needed by a process. The kernel must implement some strategy for deciding which pages to move out of memory so that it can replace these pages with the ones that are currently needed in memory. Ideally, the strategy will choose pages for replacement that will not be needed soon. An approximation to this strategy is to find pages that have not been used actively or recently.

The 4.4BSD system implemented demand paging with a page-replacement algorithm that approximated global least-recently used [Easton & Franaszek, 1979]. In FreeBSD, the one-bit use field for each page has been augmented with an activity counter to approximate global least-actively used. Both these algorithms are examples of a global replacement algorithm: one in which the choice of a page for replacement is made according to systemwide criteria. A local replacement algorithm would choose a process for which to replace a page and then chose a page based on per-process criteria. Although the algorithm in FreeBSD is similar in nature to that in 4.4BSD, its implementation is considerably different.

The kernel scans physical memory on a regular basis, considering pages for replacement. The use of a systemwide list of pages forces all processes to compete for memory on an equal basis. Note that it is also consistent with the way that FreeBSD treats other resources provided by the system. A common alternative to allowing all processes to compete equally for memory is to partition memory into multiple independent areas, each localized to a collection of processes that compete with one another for memory. This scheme is used, for example, by the VMS operating system [Kenah & Bate, 1984]. With this scheme, system administrators can guarantee that a process, or collection of processes, will always have a minimal percentage of memory. Unfortunately, this scheme can be difficult to administer. Allocating too small a number of pages to a partition can result in underutilization of memory and excessive I/O activity to secondary-storage devices, whereas setting the number too high can result in excessive swapping [Lazowska & Kelsey, 1978].

The kernel divides the main memory into five lists:

  1. Wired: Wired pages are locked in memory and cannot be paged out. Typically, these pages are being used by the kernel or the physical-memory pager, or they have been locked down with mlock. In addition, all the pages being used to hold the user structure and thread stacks of loaded (i.e., not swapped-out) processes are also wired. Wired pages cannot be paged out.

  2. Active: Active pages are being used by one or more regions of virtual memory. Although the kernel can page them out, doing so is likely to cause an active process to fault them back again.

  3. Inactive: Inactive pages have contents that are still known, but they are not usually part of any active region. If the contents of the page are dirty, the contents must be written to backing store before the page can be reused. Once the page has been cleaned, it is moved to the cache list. If the system becomes short of memory, the pageout daemon may try to move active pages to the inactive list in the hopes of finding pages that are not really in use. The selection criteria that are used by the pageout daemon to select pages to move from the active list to the inactive list are described later in this section. When the free-memory and cache lists drop too low, the pageout daemon traverses the inactive list to create more cache and free pages.

  4. Cache: Cache pages have contents that are still known, but they are not usually part of any active region. If they are mapped into an active region, they must be marked read-only so that any write attempt will cause them to be moved off the cache list. They are similar to inactive pages except that they are not dirty, either because they are unmodified, since they were paged in, or because they have been written to their backing store. They can be moved to the free list when needed.

  5. Free: Free pages have no useful contents and will be used to fulfill new page-fault requests. The idle process attempts to keep about 75 percent of the pages on the free list zeroed so that they do not have to be zeroed while servicing an anonymous-region page fault. Pages with unknown contents are placed at the front of the free list. Zeroed pages are placed at the end of the free list. The idle process takes pages from the front of the free list, zeros them, marks them as having been zeroed, and puts them on the end of the free list. Page faults that will be filling a page take one from the front of the free list. Page faults that need a zero-filled page take one from the end of the free list. If marked as zeroed, it does not have to be zeroed during the fault service.

The pages of main memory that can be used by user processes are those on the active, inactive, cache, and free lists. Requests for new pages are taken first from the free list if it has pages available otherwise from the cache list. The cache and free lists are subdivided into separate lists organized by page color.

Ideally, the kernel would maintain a working set for each process in the system. It would then know how much memory to provide to each process to minimize the latter's page-fault behavior. The FreeBSD virtual-memory system does not use the working-set model because it lacks accurate information about the reference pattern of a process. It does track the number of pages held by a process via the resident-set size, but it does not know which of the resident pages constitute the working set. In 4.3BSD, the count of resident pages was used in making decisions on whether there was enough memory for a process to be swapped in when that process wanted to run. This feature was not carried over to the FreeBSD virtual-memory system. Because it worked well during periods of high memory demand, this feature should be incorporated in future FreeBSD systems.

Paging Parameters

The memory-allocation needs of processes compete constantly, through the page-fault handler, with the overall system goal of maintaining a minimum threshold of pages in the inactive, cache, and free lists. As the system operates, it monitors main-memory utilization and attempts to run the pageout daemon frequently enough to keep the amount of inactive, cache, and free memory at or above the minimum threshold shown in Table 5.2. When the page-allocation routine, vm_page_alloc(), determines that more memory is needed, it awakens the pageout daemon.

Table 5.2. Available-memory thresholds.

Pool

Minimum

Target

Free

0.7%

3%

Cache

3%

6%

Inactive

0%

4.5%

The number of pages to be reclaimed by the pageout daemon is a function of the memory needs of the system. As more memory is needed by the system, more pages are scanned. This scanning causes the number of pages freed to increase. The pageout daemon determines the memory needed by comparing the number of available-memory pages against several parameters that are calculated during system startup. The desired values for the paging parameters are communicated to the pageout daemon through global variables that may be viewed or changed with sysctl. Likewise, the pageout daemon records its progress in global counters that may be viewed or reset with sysctl. Progress is measured by the number of pages scanned over each interval that it runs.

The goal of the pageout daemon is to maintain the inactive, cache, and free queues between the minimum and target thresholds shown in Table 5.2. The pageout daemon achieves this goal by moving pages from more active queues to less active queues to reach the indicated ranges. It moves pages from the cache list to the free list to maintain the free list minimum. It moves pages from the inactive list to the cache list to keep the cache list near its target. It moves pages from the active list to the inactive list to maintain the inactive list near its target.

The Pageout Daemon

Page replacement is done by the pageout daemon (process 2). The paging policy of the pageout daemon is embodied in the vm_pageout() and vm_pageout_scan() routines. When the pageout daemon reclaims pages that have been modified, it is responsible for writing them to the swap area. Thus, the pageout daemon must be able to use normal kernel-synchronization mechanisms, such as sleep(). It therefore runs as a separate process, with its own process structure and kernel stack. Like init, the pageout daemon is created by an internal fork operation during system startup (see Section 14.4); unlike init, however, it remains in kernel mode after the fork. The pageout daemon simply enters vm_pageout() and never returns. Unlike some other users of the disk I/O routines, the pageout process needs to do its disk operations asynchronously so that it can continue scanning in parallel with disk writes.

Historically, the pages were handled using a least-recently used algorithm. The drawback to this algorithm is that a sudden burst of memory activity can flush many useful pages from the cache. To mitigate this behavior, FreeBSD uses a least-actively used algorithm to preserve pages that have a history of usage so that they will be favored over the once-used pages brought in during a period of high memory demand.

When a page is first brought into memory it is given an initial usage count of three. Further usage information is gathered by the pageout daemon during its periodic scans of memory. As each page of memory is scanned, its reference bit is checked. If the bit is set, it is cleared and the usage counter for the page is incremented (up to a limit of 64) by the number of references to the page. If the reference bit is clear, the usage counter is decremented. When the usage counter reaches zero, the page is moved from the active list to the inactive list. Pages that are repeatedly used build up large usage counts that will cause them to remain on the active list much longer than pages that are used just once.

The goal of the pageout daemon is to keep the number of pages on the inactive, cache, and free lists within their desired ranges. Whenever an operation that uses pages causes the amount of free memory to fall below the minimum thresholds, the pageout daemon is awakened. The pageout-handling algorithm is shown in Figure 5.14 (on pages 192-193). In the following overview, the lettered points are references to the tags down the left side of the code.

  1. The pageout daemon calculates the number of pages that need to be moved from the inactive list to the cache list. As some will later be moved to the free list, enough pages must moved to the cache list to leave its minimum level after the free list has been filled. To avoid saturating the I/O system, the pageout daemon limits the number of I/O operations that it will start concurrently.

  2. Scan the inactive list until the desired number of pages are moved. Skip over busy pages, since they are likely being paged out and can be moved later when they are clean.

  3. If we find a page that is part of an object that has started being used, then it has been moved to the inactive list prematurely. So update its usage count and move it back to the active list. Pages with no associated object have no useful contents and are moved to the free list. Pages that are clean can be moved to the cache list.

  4. Dirty pages need to be paged out, but flushing a page is extremely expensive compared to freeing a clean page. Thus, dirty pages are given extra time on the inactive queue by cycling them through the queue twice before being flushed. They cycle through the list once more while being cleaned. This extra time on the inactive queue will reduce unnecessary I/O caused by prematurely paging out an active page. The clustering checks for up to eight dirty pages on either side of the selected page. The pager is only required to write the selected page. However, it may write as many of the clustered dirty pages as it finds convenient. The scanning of the inactive list stops if the number of pageouts in progress hits its limit. In 4.4BSD, the I/O completions were handled by the pageout daemon. FreeBSD requires that pagers track their own I/O operations including the appropriate updating of the written pages. The written-page update at I/O completion does not move the page from the inactive list to the cache list. Rather, the page remains on the inactive list until it is eventually moved to the cache list during a future pass of the pageout daemon.

    Figure 5.14. Pageout handling.
    /*
 * Vm_pageout_scan does the dirty work for the pageout daemon.
 */
void vm_pageout_scan(void)
{
[A] page_shortage = free_target + cache_minimum -
        (free_count + cache_count);
    max_writes_in_progress = 32;
[B] for (page = FIRST(inactive list); page; page = next) {
        next = NEXT(page);
        if (page_shortage < 0)
            break;
        if (page busy)
            continue;
[C]     if (page's object has ref_count > 0 &&
            page is referenced)
            update page active count;
            move page to end of active list;
            continue;
        }
        if (page is invalid)
            move page to front of free list;
            page shortage--;
            continue;
        }
[D]     if (first time page seen dirty) {
            mark page as seen dirty;
            move to end of inactive list;
            continue;
        }
        check for cluster of dirty pages around page;
        start asynchronous write of page cluster;
        move to end of inactive list;
        page_shortage--;
        max_writes_in_progress--;
        if (max_writes_in_progress == 0)
            break;
    }
[E] page_shortage = free_target + cache_minimum + inactive_target
        - (free_count + cache_count + inactive_count);
[F] for (page = FIRST(active list); page; page = next) {
        next = NEXT(page);
        if (page_shortage < 0)
            break;
        if (page busy)
            continue;
        if (page's object has ref_count > 0) {
            update page active count;
            move page to end of active list;
            continue;
        }
[G]     decrement page active count;
        if (page active count > 0) {
            move page to end of active list;
            continue;
[G]     decrement page active count;
        if (page active count > 0) {
            move page to end of active list;
            continue;
        }
        page_shortage--;
        if (page is clean)
            move page to end of cache list;
        else
            move page to end of inactive list;
    }
[H] page_shortage = free_minimum - free_count;
    for (page = FIRST(cache list); page; page = next) {
        next = NEXT(page);
        if (page_shortage < 0)
            break;
        if (page busy)
            continue;
        move page to front of free list;
    }
[I] if (targets not met)
        request swap-out daemon to run;
[J] if (nearly all memory and swap in use)
        kill biggest process;
}

  5. The pageout daemon calculates the number of pages that need to be moved from the active list to the inactive list. As some will eventually need to be moved to the cache and free lists, enough pages must be moved to the inactive list to leave it at its target level after the cache and free lists have been filled.

  6. Scan the active list until the desired number of pages are moved. Skip over busy pages as they are likely being actively used. If we find a page that is part of an active object, update its usage count and move it to the end of the active list.

  7. The page is not active, so decrement its usage count. If its usage is still above zero, move it to the end of the active list. Otherwise, move it to the inactive list if it is dirty or the cache list if it is clean.

  8. Calculate the number of pages that need to be moved to the free list. It only fills the free list to its minimum level. It prefers to keep pages on the cache list since they can be reclaimed if their object or file is reactivated. Once on the free list the identity of a page is lost. Thus, the pageout daemon delays moving pages to the free list until just before they are expected to be needed.

  9. If the page-count targets have not been met, the swap-out daemon is started (see next subsection) to try to clear additional memory.

  10. The FreeBSD 5.2 kernel does not impose any limits on the amount of virtual memory that it will grant. If it finds that it has nearly filled its memory and swap space, it avoids going into deadlock by killing off the largest process. One day virtual-memory accounting and limits will be added to the system so as to avoid this crude resource-control mechanism.

Note that the page and object locking has been elided in Figure 5.14 to simplify the explanation.

Swapping

Although swapping is generally avoided, there are several times when it is used in FreeBSD to address a serious memory shortage. Swapping is done in FreeBSD when any of the following occurs:

  • The system becomes so short of memory that the paging process cannot free memory fast enough to satisfy the demand. For example, a memory shortfall may happen when multiple large processes are run on a machine lacking enough memory for the minimum working sets of the processes.

  • Processes are completely inactive for more than 10 seconds. Otherwise, such processes would retain a few pages of memory associated with their user structure and thread stacks.

Swap operations completely remove a process from main memory, including the process page tables, the pages of the data and the stack segments that are not already in swap space, and the user structure and thread stacks.

Process swapping is invoked only when paging is unable to keep up with memory needs or when short-term memory needs warrant swapping a process. In general, the swap-scheduling mechanism does not do well under heavy load; system performance is much better when memory scheduling can be done by the page-replacement algorithm than when the swap algorithm is used.

Swap out is driven by the swap-out daemon, vmdaemon (process 3). The swap-out policy of the vmdaemon is embodied in the vm_daemon() routine. If the pageout daemon can find any processes that have been sleeping for more than 10 seconds (swap_idle_threshold2, the cutoff for considering the time sleeping to be "a long time"), it will swap out the one sleeping for the longest time. Such processes have the least likelihood of making good use of the memory that they occupy; thus, they are swapped out even if they are small. If none of these processes are available, the pageout daemon will swap out a process that has been sleeping for as briefly as 2 seconds (swap_idle_thresholdl). These criteria attempt to avoid swapping entirely until the pageout daemon is clearly unable to keep enough memory free.

In 4.4BSD, if memory was still desperately low, the swap-out daemon would select to swap out the runnable process that had been resident the longest. Once swapping of runnable processes had begun, the processes eligible for swapping would take turns in memory so that no process was frozen out entirely. The FreeBSD swap-out daemon will not select a runnable processes to swap out. So, if the set of runnable processes do not fit in memory, the machine will effectively deadlock. Current machines have enough memory that this condition does not arise. If it does, the 4.4BSD algorithm will need to be restored.

The mechanics of doing a swap out are simple. The swapped-in process flag P_INMEM is cleared to show that the process is not resident in memory. The PS_SWAPPINGOUT flag is set while the swap out is being done so that neither a second swap out nor a swap in is attempted at the same time. If a runnable process is to be swapped (which currently never happens), it needs to be removed from the runnable process queue. The user structure of the process and the kernel stack of its threads are then marked as pageable, which allows the user structure and stack pages, along with any other remaining pages for the process, to be paged out via the standard pageout mechanism. The swapped-out process cannot be run until after it is swapped back into memory.

The Swap-In Process

Swap-in operations are done by the swapping process, swapper (process 0). This process is the first one created by the system when the latter is started. The swap-in policy of the swapper is embodied in the scheduler() routine. This routine swaps processes back in when memory is available and they are ready to run. At any time, the swapper is in one of three states:

  1. Idle: No swapped-out processes are ready to be run. Idle is the normal state.

  2. Swapping in: At least one runnable process is swapped out, and scheduler() attempts to find memory for it.

  3. Swapping out: The system is short of memory, or there is not enough memory to swap in a process. Under these circumstances, scheduler() awakens the pageout daemon to free pages and to swap out other processes until the memory shortage abates.

If more than one swapped-out process is runnable, the first task of the swapper is to decide which process to swap in. This decision may affect the decision whether to swap out another process. Each swapped-out process is assigned a priority based on

  • The length of time it has been swapped out

  • Its nice value

  • The amount of time it was asleep since it last ran

In general, the process that has been swapped out longest or was swapped out because it had slept for a long time before being swapped will be brought in first. Once a process is selected, the swapper checks to see whether there is enough memory free to swap in the process. Historically, the system required as much memory to be available as was occupied by the process before that process was swapped. Under FreeBSD, this requirement was reduced to a requirement that the number of pages on the free and cache lists be at least equal to the minimum free-memory threshold. If there is enough memory available, the process is brought back into memory. The user structure is swapped in immediately, but the process loads the rest of its working set by demand paging from the backing store. Thus, not all the memory that is needed by the process is used immediately. Earlier BSD systems tracked the anticipated demand and would only swap in runnable processes as free memory became available to fulfill their expected needs. FreeBSD allows all swapped-out runnable processes to be swapped in as soon as there is enough memory to load their user structure and thread stacks.

The procedure for swap in of a process is the reverse of that for swap out:

1. Memory is allocated for the user structure and the kernel stack of each of its threads, and they are read back from swap space.

2. The process is marked as resident, and its runnable threads are returned to the run queue (i.e., those threads that are not stopped or sleeping).

After the swap in completes, the process is ready to run like any other, except that it has no resident pages. It will bring in the pages that it needs by faulting them.


5.13. Portability

Everything discussed in this chapter up to this section has been part of the machine-independent data structures and algorithms. These parts of the virtual-memory system require little change when FreeBSD is ported to a new architecture. This section will describe the machine-dependent parts of the virtual-memory system; the parts of the virtual-memory system that must be written as part of a port of FreeBSD to a new architecture. The machine-dependent parts of the virtual-memory system control the hardware memory-management unit (MMU). The MMU implements address translation and access control when virtual memory is mapped onto physical memory.

One common MMU design uses memory-resident forward-mapped page tables. These page tables are large contiguous arrays indexed by the virtual address. There is one element, or page-table entry, in the array for each virtual page in the address space. This element contains the physical page to which the virtual page is mapped, as well as access permissions, status bits telling whether the page has been referenced or modified, and a bit showing whether the entry contains valid information. For a 4-Gbyte address space with 4-Kbyte virtual pages and a 32-bit page-table entry, 1 million entries, or 4 Mbyte, would be needed to describe an entire address space. Since most processes use little of their address space, most of the entries would be invalid, and allocating 4 Mbyte of physical memory per process would be wasteful. Thus, most page-table structures are hierarchical, using two or more levels of mapping. With a hierarchical structure, different portions of the virtual address are used to index the various levels of the page tables. The intermediate levels of the table contain the addresses of the next lower level of the page table. The kernel can mark as unused large contiguous regions of an address space by inserting invalid entries at the higher levels of the page table, eliminating the need for invalid page descriptors for each individual unused virtual page.

This hierarchical page-table structure requires the hardware to make frequent memory references to translate a virtual address. To speed the translation process, most page-table-based MMUs also have a small, fast, fully associative hardware cache of recent address translations, a structure known commonly as a translation lookaside buffer (TLB). When a memory reference is translated, the TLB is first consulted and, only if a valid entry is not found there, the page-table structure for the current process is traversed. Because most programs exhibit spatial locality in their memory-access patterns, the TLB does not need to be large; many are as small as 128 entries.

As address spaces grew beyond 32 to 48 and, more recently, 64 bits, simple indexed data structures become unwieldy, with three or more levels of tables required to handle address translation. A response to this page-table growth is the inverted page table, also known as the reverse-mapped page table. In an inverted page table, the hardware still maintains a memory-resident table, but that table contains one entry per physical page and is indexed by physical address instead of by virtual address. An entry contains the virtual address to which the physical page is currently mapped, as well as protection and status attributes. The hardware does virtual-to-physical address translation by computing a hash function on the virtual address to select an entry in the table. The system handles collisions by linking together table entries and making a linear search of this chain until it finds the matching virtual address.

The advantages of an inverted page table are that the size of the table is proportional to the amount of physical memory and that only one global table is needed, rather than one table per process. A disadvantage to this approach is that there can be only one virtual address mapped to any given physical page at any one time. This limitation makes virtual-address aliasing—having multiple virtual addresses for the same physical page—difficult to handle. As it is with the forward-mapped page table, a hardware TLB is typically used to speed the translation process.

A final common MMU organization consists of just a TLB. This architecture is the simplest hardware design. It gives the software maximum flexibility by allowing the latter to manage translation information in whatever structure it desires.

Often, a port to another architecture with a similar memory-management organization can be used as a starting point for a new port. The PC architecture uses the typical two-level page-table organization shown in Figure 5.15. An address space is broken into 4-Kbyte virtual pages, with each page identified by a 32-bit entry in the page table. Each page-table entry contains the physical page number assigned to the virtual page, the access permissions allowed, modify and reference information, and a bit showing that the entry contains valid information. The 4 Mbyte of page-table entries are likewise divided into 4-Kbyte page-table pages, each of which is described by a single 32-bit entry in the directory table. Directory-table entries are nearly identical to page-table entries: They contain access bits, modify and reference bits, a valid bit, and the physical page number of the page-table page described. One 4-Kbyte page—1024 directory-table entries—covers the maximum-sized 4-Gbyte address space. The CR3 hardware register contains the physical address of the directory table for the currently active process.

Figure 5.15. Two-level page-table organization. Key: V—page-valid bit; M—page-modified bit; R—page-referenced bit; ACC—page-access permissions.

[View full size image]

In Figure 5.15, translation of a virtual address to a physical address during a CPU access proceeds as follows:

1. The 10 most significant bits of the virtual address are used to index into the active directory table.

2. If the selected directory-table entry is valid and the access permissions grant the access being made, the next 10 bits of the virtual address are used to index into the page-table page referenced by the directory-table entry.

3. If the selected page-table entry is valid and the access permissions match, the final 12 bits of the virtual address are combined with the physical page referenced by the page-table entry to form the physical address of the access.

The Role of the pmap Module

The machine-dependent code describes how the physical mapping is done between the user-processes and kernel virtual addresses and the physical addresses of the main memory. This mapping function includes management of access rights in addition to address translation. In FreeBSD, the physical-mapping (pmap) module manages machine-dependent translation and access tables that are used either directly or indirectly by the memory-management hardware. For example, on the PC, the pmap maintains the memory-resident directory and page tables for each process, as well as for the kernel. The machine-dependent state required to describe the translation and access rights of a single page is often referred to as a mapping or mapping structure.

The FreeBSD pmap interface is nearly identical to that in Mach 3.0 and it shares many design characteristics. The pmap module is intended to be logically independent of the higher levels of the virtual-memory system. The interface deals strictly in machine-independent page-aligned virtual and physical addresses and in machine-independent protections. The machine-independent page size may be a multiple of the architecture-supported page size. Thus, pmap operations must be able to affect more than one physical page per logical page. The machine-independent protection is a simple encoding of read, write, and execute permission bits. The pmap must map all possible combinations into valid architecture-specific values.

A process's pmap is considered to be a cache of mapping information kept in a machine-dependent format. As such, it does not need to contain complete state for all valid mappings. Mapping state is the responsibility of the machine-independent layer. With one exception, the pmap module may throw away mapping state at its discretion to reclaim resources. The exception is wired mappings, which should never cause a fault that reaches the machine-independent vm_fault() routine. Thus, state for wired mappings must be retained in the pmap until it is removed explicitly.

In general, pmap routines may act either on a set of mappings defined by a virtual address range or on all mappings for a particular physical address. Being able to act on individual or all virtual mappings for a physical page requires that the mapping information maintained by the pmap module be easily found by both virtual and physical address. For architectures such as the PC that support memory-resident page tables, the virtual-to-physical, or forward lookup, may be a simple emulation of the hardware page-table traversal. Physical-to-virtual, or reverse, lookup uses a list of pv_entry structures, described in the next subsection, to find all the page-table entries referencing a page. The list may contain multiple entries only if virtual-address aliasing is allowed.

There are two strategies that can be used for management of pmap memory resources, such as user-directory or page-table memory. The traditional and easiest approach is for the pmap module to manage its own memory. Under this strategy, the pmap module can grab a fixed amount of wired physical memory at system boot time, map that memory into the kernel's address space, and allocate pieces of the memory as needed for its own data structures. The primary benefit is that this approach isolates the pmap module's memory needs from those of the rest of the system and limits the pmap module's dependencies on other parts of the system. This design is consistent with a layered model of the virtual-memory system in which the pmap is the lowest, and hence self-sufficient, layer.

The disadvantage is that this approach requires the duplication of many of the memory-management functions. The pmap module has its own memory allocator and deallocator for its private heap—a heap that is statically sized and cannot be adjusted for varying systemwide memory demands. For an architecture with memory-resident page tables, it must keep track of noncontiguous chunks of processes' page tables, because a process may populate its address space sparsely. Handling this requirement entails duplicating much of the standard list-management code, such as that used by the vm_map code.

An alternative approach, used by the PC, is to use the higher-level virtual-memory code recursively to manage some pmap resources. Here, the 4-Kbyte directory table for each user process is mapped into the address space of the kernel as part of setting up the process and remains resident until the process exits. While a process is running, its page-table entries are mapped into a virtually contiguous 4-Mbyte array of page-table entries in the kernel's address space. This organization leads to an obscure memory-saving optimization exploited in the PC pmap module where the kernel's page-table page describing the 4-Mbyte user page-table range can double as the user's directory table. The kernel also maintains alternate maps to hold individual page-table pages of other nonrunning processes if it needs to access their address space.

Using the same page-allocation routines as all the other parts of the system ensures that physical memory is allocated only when needed and from the systemwide free-memory pool. Page tables and other pmap resources also can be allocated from pageable kernel memory. This approach easily and efficiently supports large sparse address spaces, including the kernel's own address space.

The primary drawback is that this approach violates the independent nature of the interface. In particular, the recursive structure leads to deadlock problems with global multiprocessor spin locks that can be held while the kernel is calling a pmap routine.

The pmap data structures are contained in the machine-dependent include directory in the file pmap.h. Most of the code for these routines is in the machine-dependent source directory in the file pmap.c. The main tasks of the pmap module are these:

  • System initialization and startup (pmap_bootstrap(), pmap_init(), pmap_growkernel())

  • Allocation and deallocation of mappings of physical to virtual pages (pmap_enter(), pmap_remove(), pmap_qenter(), pmap_qremove())

  • Change of access protections and other attributes of mappings (pmap_change_wiring(), pmap_page_protect(), pmap_protect())

  • Maintenance of physical page-usage information (pmap_clear_modify(), pmap_clear_reference(), pmap_is_modified(), pmap_ts_referenced())

  • Initialization of physical pages (pmap_copy_page(), pmap_zero_page())

  • Management of internal data structures (pmap_pinit(), pmap_release())

Each of these tasks is described in the following subsections.

Initialization and Startup

The first step in starting up the system is for the loader to bring the kernel image from a disk or the network into the physical memory of the machine. The kernel load image looks much like that of any other process; it contains a text segment, an initialized data segment, and an uninitialized data segment. The loader places the kernel contiguously into the beginning of physical memory. Unlike a user process that is demand paged into memory, the text and data for the kernel are read into memory in their entirety. Following these two segments, the loader zeros an area of memory equal to the size of the kernel's uninitialized memory segment. After loading the kernel, the loader passes control to the starting address given in the kernel executable image. When the kernel begins executing, it is executing with the MMU turned off. Consequently, all addressing is done using the direct physical addresses.

The first task undertaken by the kernel is to set up the kernel pmap, and any other data structures that are necessary to describe the kernel's virtual address space. On the PC, the initial setup includes allocating and initializing the directory and page tables that map the statically loaded kernel image and memory-mapped I/O address space, allocating a fixed amount of memory for kernel page-table pages, allocating and initializing the user structure and kernel stack for the initial process, reserving special areas of the kernel's address space, and initializing assorted critical pmap-internal data structures. When done, it is possible to enable the MMU. Once the MMU is enabled, the kernel begins running in the context of process zero.

Once the kernel is running in its virtual address space, it proceeds to initialize the rest of the system. It determines the size of the physical memory, then calls pmap_bootstrap() and vm_page_startup() to set up the initial pmap data structures, to allocate the vm_page structures, and to create a small, fixed-size pool of memory, which the kernel memory allocators can use so that they can begin responding to memory allocation requests. Next it makes a call to set up the machine-independent portion of the virtual-memory system. It concludes with a call to pmap_init(), which allocates all resources necessary to manage multiple user address spaces and synchronizes the higher-level kernel virtual-memory data structures with the kernel pmap.

Pmap_init() allocates a minimal amount of wired memory to use for kernel page-table pages. The page-table space is expanded dynamically by the pmap_growkernel() routine as it is needed while the kernel is running. Once allocated it is never freed. The limit on the size of the kernel's address space is selected at boot time. On the PC, the kernel is typically given a maximum of 1 Gbyte of address space.

In 4.4BSD, the memory managed by the buffer cache was separate from the memory managed by the virtual-memory system. Since all the virtual-memory pages were used to map process regions, it was sensible to create an inverted page table. This table was an array of pv_entry structures. Each pv_entry described a single address translation and included the virtual address, a pointer to the associated pmap structure for that virtual address, a link for chaining together multiple entries mapping this physical address, and additional information specific to entries mapping page-table pages. Building a dedicated table was sensible, since all valid pages were referenced by a pmap, yet few had multiple mappings.

With the merger of the buffer cache into the virtual-memory system in FreeBSD, many pages of memory are used to cache file data that is not mapped into any process address space. Thus, preallocating a table of pv_entry structures is wasteful, since many of them would go unused. So, FreeBSD allocates pv_entry structures on demand as pages are mapped into a process address space.

Figure 5.16 shows the pv_entry references for a set of pages that have a single mapping. The purpose of the pv_entry structures is to identify the address space that has the page mapped. The machine-dependent part of each vm_page structure contains the head of list of pv_entry structures and a count of the number of entries on the list. In Figure 5.16, the object is using pages 5, 18, and 79. The list heads in the machine-dependent structures of these vm_page structures would each point to a single pv_entry structure labelled in the figure with the number of the vm_page structure that references them. Not shown in Figure 5.16 is that each physical map structure also maintains a list of all the pv_entry structures that reference it.

Figure 5.16. Physical pages with a single mapping.

[View full size image]

Each pv_entry can reference only one physical map. When an object becomes shared between two or more processes, each physical page of memory becomes mapped into two or more sets of page tables. To track these multiple references, the pmap module must create chains of pv_entry structures, as shown in Figure 5.17. Copy-on-write is an example of the need to find all the mappings of a page as it requires that the page tables be set to read-only in all the processes sharing the object. The pmap module can implement this request by walking the list of pages associated with the object to be made copy-on-write. For each page, it traverses that page's list of pv_entry structures. It then makes the appropriate change to the page-table entry associated with each pv_entry structure.

Figure 5.17. Physical pages with multiple mappings.

[View full size image]

A system with many shared objects can require many pv_entry structures, which can use an unreasonable amount of the kernel memory. The alternative would be to keep a list associated with each object of all the vm_map_entry structures that reference it. When it becomes necessary to modify the mapping of all the references to the page, the kernel could traverse this list checking the address space associated with each vm_map_entry to see if it held a reference to the page. For each page found, it could make the appropriate update.

The pv_entry structures consume more memory but reduce the time to do a common operation. For example, consider a system running a thousand processes that all share a common library. Without the pv_entry list, the cost to change a page to copy-on-write would require checking all thousand processes. With the pv_entry list, only those processes using the page would need to be inspected.

Mapping Allocation and Deallocation

The primary responsibility of the pmap module is validating (allocating) and invalidating (deallocating) mappings of physical pages to virtual addresses. The physical pages represent cached portions of an object that is providing data from a file or an anonymous memory region. A physical page is bound to a virtual address because that object is being mapped into a process's address space either explicitly by mmap or implicitly by fork or exec. Physical-to-virtual address mappings are not created at the time that the object is mapped; instead, their creation is delayed until the first reference to a particular page is made. At that point, an access fault will occur, and pmap_enter() will be called. Pmap_enter(), is responsible for any required side effects associated with creation of a new mapping. Such side effects are largely the result of entering a second translation for an already mapped physical page—for example, as the result of a copy-on-write operation. Typically, this operation requires flushing uniprocessor or multiprocessor TLB or cache entries to maintain consistency.

In addition to its use to create new mappings, pmap_enter() may also be called to modify the wiring or protection attributes of an existing mapping or to rebind an existing mapping for a virtual address to a new physical address. The kernel can handle changing attributes by calling the appropriate interface routine, described in the next subsection. Changing the target physical address of a mapping is simply a matter of first removing the old mapping and then handling it like any other new mapping request.

Pmap_enter() is the only routine that cannot lose state or delay its action. When called, it must create a mapping as requested, and it must validate that mapping before returning to the caller. On the PC, pmap_enter() must first check whether a page-table entry exists for the requested address. If a physical page has not yet been allocated to the process page-table at the location required for the new mapping, a zeroed page is allocated, wired, and inserted into the directory table of the process.

After ensuring that all page-table resources exist for the mapping being entered, pmap_enter() validates or modifies the requested mapping as follows:

1. Check to see whether a mapping structure already exists for this virtual-to-physical address translation. If one does, the call must be one to change the protection or wiring attributes of the mapping; it is handled as described in the next subsection.

2. Otherwise, if a mapping exists for this virtual address but it references a different physical address, that mapping is removed.

3. The hold count on a page-table page is incremented each time a new page reference is added and decremented each time an old page reference is removed. When the last valid page is removed, the hold count drops to zero, the page is unwired, and the page-table page is freed because it contains no useful information.

4. A page-table entry is created and validated, with cache and TLB entries flushed as necessary.

5. If the physical address is outside the range managed by the pmap module (e.g., a frame-buffer page), no pv_entry structure is needed. Otherwise, for the case of a new mapping for a physical page that is mapped into an address space, a pv_entry structure is created.

6. For machines with a virtually-indexed cache, a check is made to see whether this physical page already has other mappings. If it does, all mappings may need to be marked cache inhibited, to avoid cache inconsistencies.

When an object is unmapped from an address space, either explicitly by munmap or implicitly on process exit, the pmap module is invoked to invalidate and remove the mappings for all physical pages caching data for the object. Unlike pmap_enter(), pmap_remove() can be called with a virtual-address range encompassing more than one mapping. Hence, the kernel does the unmapping by looping over all virtual pages in the range, ignoring those for which there is no mapping and removing those for which there is one.

Pmap_remove() on the PC is simple. It loops over the specified address range, invalidating individual page mappings. Since pmap_remove() can be called with large sparsely allocated regions, such as an entire process virtual address range, it needs to skip efficiently invalid entries within the range. It skips invalid entries by first checking the directory-table entry for a particular address and, if an entry is invalid, skipping to the next 4-Mbyte boundary. When all page mappings have been invalidated, any necessary global cache flushing is done.

To invalidate a single mapping, the kernel locates and marks as invalid the appropriate page-table entry. The reference and modify bits for the page are saved in the page's vm_page structure for future retrieval. If this mapping was a user mapping, the hold count for the page-table page is decremented. When the count reaches zero, the page-table page can be reclaimed because it contains no more valid mappings. When a user page-table page is removed from the kernel's address space (i.e., as a result of removal of the final valid user mapping from that page), the process's directory table must be updated. The kernel does this update by invalidating the appropriate directory-table entry. If the physical address from the mapping is outside the managed range, nothing more is done. Otherwise, the pv_entry structure is found and is deallocated.

The pmap_qenter() and pmap_qremove() are faster versions of the pmap_enter() and pmap_remove() functions that can be used by the kernel to quickly create and remove temporary mappings. They can only be used on non-pageable mappings in the address space of the kernel. For example, the buffer-cache management routines use these routines to map file pages into kernel memory so that they can be read or written by the filesystem.

Change of Access and Wiring Attributes for Mappings

An important role of the pmap module is to manipulate the hardware access protections for pages. These manipulations may be applied to all mappings covered by a virtual-address range within a pmap via pmap_protect(), or they may be applied to all mappings of a particular physical page across pmaps via pmap_page_protect(). There are two features common to both calls. First, either form may be called with a protection value of VM_PROT_NONE to remove a range of virtual addresses or to remove all mappings for a particular physical page. Second, these routines should never add write permission to the affected mappings. Thus, calls including VM_PROT_WRITE should make no changes. This restriction is necessary for the copy-on-write mechanism to function properly. The request to make the page writable is made only in the vm_map_entry structure. When a later write attempt on the page is made by the process, a page fault will occur. The page-fault handler will inspect the vm_map_entry and determine that the write should be permitted. If it is a copy-on-write page, the fault handler will make any necessary copies before calling pmap_enter() to enable writing on the page. Thus, write permission on a page is added only via calls to pmap_enter().

Pmap_protect() is used primarily by the mprotect system call to change the protection for a region of process address space. The strategy is similar to that of pmap_remove(): Loop over all virtual pages in the range and apply the change to all valid mappings that are found. Invalid mappings are left alone.

For the PC, pmap_protect() first checks for the special cases. If the requested permission is VM_PROT_NONE, it calls pmap_remove() to handle the revocation of all access permission. If VM_PROT_WRITE is included, it just returns immediately. For a normal protection value, pmap_remove() loops over the given address range, skipping invalid mappings. For valid mappings, the page-table entry is looked up, and, if the new protection value differs from the current value, the entry is modified and any TLB and cache flushing done. As occurs with pmap_remove(), any global cache actions are delayed until the entire range has been modified.

Pmap_page_protect() is used internally by the virtual-memory system for two purposes. It is called to set read-only permission when a copy-on-write operation is set up (e.g., during fork). It also removes all access permissions before doing page replacement to force all references to a page to block pending the completion of its operation. In Mach, this routine used to be two separate routines—pmap_clear_ptes() and pmap_remove_all()—and many pmap modules implement pmap_page_protect() as a call to one or the other of these functions, depending on the protection argument.

In the PC implementation of pmap_page_protect(), if VM_PROT_WRITE is requested, it returns without doing anything. The addition of write enable must be done on a page-by-page basis by the page-fault handling routine as described for pmap_protect(). Otherwise, it traverses the list of pv_entry structures for this page, invalidating the individual mappings as described in the previous subsection. As occurs with pmap_protect(), the entry is checked to ensure that it is changing before expensive TLB and cache flushes are done. Note that TLB and cache flushing differ from those for pmap_remove(), since they must invalidate entries from multiple process contexts, rather than invalidating multiple entries from a single process context.

Pmap_change_wiring() is called to wire or unwire a single machine-independent virtual page within a pmap. As described in the previous subsection, wiring informs the pmap module that a mapping should not cause a hardware fault that reaches the machine-independent vm_fault() code. Wiring is typically a software attribute that has no effect on the hardware MMU state: It simply tells the pmap not to throw away state about the mapping. As such, if a pmap module never discards state, then it is not strictly necessary for the module even to track the wired status of pages. The only side effect of not tracking wiring information in the pmap is that the mlock system call cannot be completely implemented without a wired page-count statistic.

The PC pmap implementation maintains wiring information. An unused bit in the page-table-entry structure records a page's wired status. Pmap_change_wiring() sets or clears this bit when it is invoked with a valid virtual address. Since the wired bit is ignored by the hardware, there is no need to modify the TLB or cache when the bit is changed.

Management of Page-Usage Information

The machine-independent page-management code needs to be able to get basic information about the usage and modification of pages from the underlying hardware. The pmap module facilitates the collection of this information without requiring the machine-independent code to understand the details of the mapping tables by providing a set of interfaces to query and clear the reference and modify bits. The pageout daemon can call vm_page_test_dirty() to determine whether a page is dirty. If the page is dirty, the pageout daemon can write it to backing store and then call pmap_clear_modify() to clear the modify bit. Similarly, when the pageout daemon pages out or inactivates a page, it uses pmap_clear_reference() to clear the reference bit for the page. When it wants to update the active count for a page, it uses pmap_ts_referenced() to count the number of uses of the page since it was last scanned.

One important feature of the query routines is that they should return valid information even if there are currently no mappings for the page in question. Thus, referenced and modified information cannot just be gathered from the hardware-maintained bits of the various page-table or TLB entries; rather, there must be a place where the information is retained when a mapping is removed.

For the PC, the modified information for a page is stored in the dirty field of its vm_page structure. Initially cleared, the information is updated whenever a mapping for a page is considered for removal. The vm_page_test_dirty() routine first checks the dirty field and, if the bit is set, returns TRUE immediately. Since this attribute array contains only past information, it still needs to check status bits in the page-table entries for currently valid mappings of the page. This information is checked by calling the pmap_is_modified() routine, which immediately returns FALSE if it is not passed a managed physical page. Otherwise, pmap_is_modified() traverses the pv_entry structures associated with the physical page, examining the modified bit for the pv_entry's associated page-table entry. It can return TRUE as soon as it encounters a set bit or FALSE if the bit is not set in any page-table entry.

The referenced information for a page is stored in the act_count field and as a flag of its vm_page structure. Initially cleared, the information is updated periodically by the pageout daemon. As it scans memory, the pageout daemon calls the pmap_ts_referenced() routine to collect a count of references to the page. The pmap_ts_referenced() routine returns zero if it is not passed a managed physical page. Otherwise, it traverses the pv_entry structures associated with the physical page, examining and clearing the referenced bit for the pv_entry's associated page-table entry. It returns the number of referenced bits that it found.

The clear routines also return immediately if they are not passed a managed physical page. Otherwise, the referenced or modified bit is cleared in the attribute array, and they loop over all pv_entry structures associated with the physical page, clearing the hardware-maintained page-table-entry bits. This final step may involve TLB or cache flushes along the way or afterward.

Initialization of Physical Pages

Two interfaces are provided to allow the higher-level virtual-memory routines to initialize physical memory. Pmap_zero_page() takes a physical address and fills the page with zeros. Pmap_copy_page() takes two physical addresses and copies the contents of the first page to the second page. Since both take physical addresses, the pmap module will most likely have to first map those pages into the kernel's address space before it can access them.

The PC implementation has a pair of global kernel virtual addresses reserved for zeroing and copying pages. Pmap_zero_page() maps the specified physical address into the reserved virtual address, calls bzero() to clear the page, and then removes the temporary mapping with the single translation-invalidation primitive used by pmap_remove(). Similarly, pmap_copy_page() creates mappings for both physical addresses, uses bcopy() to make the copy, and then removes both mappings.

Management of Internal Data Structures

The remaining pmap interface routines are used for management and synchronization of internal data structures. Pmap_pinit() creates an instance of the machine-dependent pmap structure. It is used by the vmspace_fork() and vmspace_exec() routines when creating new address spaces during a fork or exec. Pmap_release() deallocates the pmap's resources. It is used by the vmspace_free() routine when cleaning up a vmspace when a process exits.


Exercises

5.1 What does it mean for a machine to support virtual memory? What four hardware facilities are typically required for a machine to support virtual memory?

5.2 What is the relationship between paging and swapping on a demand-paged virtual-memory system? Explain whether it is desirable to provide both mechanisms in the same system. Can you suggest an alternative to providing both mechanisms?

5.3 What three policies characterize paging systems?

5.4 What is copy-on-write? In most UNIX applications, the fork system call is followed almost immediately by an exec system call. Why does this behavior make it particularly attractive to use copy-on-write in implementing fork?

5.5 Explain why the vfork system call will always be more efficient than a clever implementation of the fork system call.

5.6 When a process exits, all its pages may not be placed immediately on the memory free list. Explain this behavior.

5.7 Why does the kernel have both the traditional malloc() and free() interface and the zone allocator? Explain when each type of interface is useful.

5.8 What is the purpose of page coloring? On what sort of hardware is it needed?

5.9 What purpose does the pageout-daemon process serve in the virtual-memory system?

5.10 What is clustering? Where is it used in the virtual-memory system?

5.11 Why is the historic use of the sticky bit to lock a process image in memory no longer useful in FreeBSD?

5.12 Give two reasons for swapping to be initiated.

*5.13 The 4.3BSD virtual-memory system had a text cache that retained the identity of text pages from one execution of a program to the next. How does the caching of vnode objects in FreeBSD improve on the performance of the 4.3BSD text cache?

**5.14 FreeBSD reduces the length of shadow chains by checking at each copy-on-write fault whether the object taking the fault completely shadows the object below it in the chain. If it does, a collapse can be done. One alternative would be to calculate the number of live references to a page after each copy-on-write fault and if only one reference remains, to move that page to the object that references it. When the last page is removed, the chain can be collapsed. Implement this algorithm and compare its cost to the current algorithm.

**5.15 The pv_entry structures could be replaced by keeping a list associated with each object of all the vm_map_entry structures that reference it. If each vm_map_entry structure had only a single list pointer in it, only the final object would be able to reference it. Shadow objects would have to find their final object to find their referencing vm_map_entry structure. Implement an algorithm to find all the references to the pages of a shadow object using this scheme. Compare its cost with that of the current algorithm using pv_entry structures.

**5.16 Port the code from 4.3BSD that would forcibly swap out runnable processes when the paging rate gets too high. Run three or more processes that each have a working set of 40 percent of the available memory. Compare the performance of this benchmark using the 4.3BSD algorithm and the current algorithm.


References

Babao [image] lu & Joy, 1981. Ö. Babaolu & W. N. Joy, "Converting a Swap-Based System to Do Paging in an Architecture Lacking Page-Referenced Bits," Proceedings of the Eighth Symposium on Operating Systems Principles, pp. 78-86, December 1981.

Bansal & Modha, 2004. S. Bansal & D. Modha, "CAR: Clock with Adaptive Replacement," Proceedings of the Third Usenix Conference on File and Storage Technologies, pp. 187-200, April 2004.

Belady, 1966. L. A. Belady, "A Study of Replacement Algorithms for Virtual Storage Systems," IBM Systems Journal, vol. 5, no. 2, pp. 78-101, 1966.

Bonwick, 1994. J. Bonwick, "The Slab Allocator: An Object-Caching Kernel Memory Allocator," Proceedings of the 1994 Usenix Annual Technical Conference, pp. 87-98, June 1994.

Bonwick & Adams, 2001. J. Bonwick & J. Adams, "Magazines and Vmem: Extending the Slab Allocator to Many CPUs and Arbitrary Resources," Proceedings of the 2001 Usenix Annual Technical Conference, pp. 15-34, June 2001.

Coffman & Denning, 1973. E. G. Coffman, Jr. & P. J. Denning, Operating Systems Theory, p. 243, Prentice-Hall, Englewood Cliffs, NJ, 1973.

Denning, 1970. P. J. Denning, "Virtual Memory," Computer Surveys, vol. 2, no. 3, pp. 153-190, September 1970.

Easton & Franaszek, 1979. M. C. Easton & P. A. Franaszek, "Use Bit Scanning in Replacement Decisions," IEEE Transactions on Computing, vol. 28, no. 2, pp. 133-141, February 1979.

Gingell et al., 1987. R. Gingell, M. Lee, X. Dang, & M. Weeks, "Shared Libraries in SunOS," USENIX Association Conference Proceedings, pp. 131-146, June 1987.

Gingell, Moran, & Shannon, 1987. R. Gingell, J. Moran, & W. Shannon, "Virtual Memory Architecture in SunOS," USENIX Association Conference Proceedings, pp. 81-94, June 1987.

Intel, 1984. Intel, "Introduction to the iAPX 286," Order Number 210308, Intel Corporation, Santa Clara, CA, 1984.

Kenah & Bate, 1984. L. J. Kenah & S. F. Bate, VAX/VMS Internals and Data Structures, Digital Press, Bedford, MA, 1984.

King, 1971. W. F. King, "Analysis of Demand Paging Algorithms," IFIP, pp. 485-490, North Holland, Amsterdam, 1971.

Korn & Vo, 1985. D. Korn & K. Vo, "In Search of a Better Malloc," USENIX Association Conference Proceedings, pp. 489-506, June 1985.

Lazowska & Kelsey, 1978. E. D. Lazowska & J. M. Kelsey, "Notes on Tuning VAX/VMS.," Technical Report 78-12-01, Department of Computer Science, University of Washington, Seattle, WA, December 1978.

Marshall, 1979. W. T. Marshall, "A Unified Approach to the Evaluation of a Class of `Working Set Like' Replacement Algorithms," PhD Thesis, Department of Computer Engineering, Case Western Reserve University, Cleveland, OH, May 1979.

McKusick & Karels, 1988. M. K. McKusick & M. Karels, "Design of a General Purpose Memory Allocator for the 4.3BSD UNIX Kernel," USENIX Association Conference Proceedings, pp. 295-304, June 1988.

Organick, 1975. E. I. Organick, The Multics System: An Examination of Its Structure, MIT Press, Cambridge, MA, 1975.

Tevanian, 1987. A. Tevanian, "Architecture-Independent Virtual Memory Management for Parallel and Distributed Environments: The Mach Approach," Technical Report CMU-CS-88-106, Department of Computer Science, Carnegie-Mellon University, Pittsburgh, PA, December 1987.

Young, 1989. M. W. Young, Exporting a User Interface to Memory Management from a Communication-Oriented Operating System, CMU-CS-89-202, Department of Computer Science, Carnegie-Mellon University, November 1989.


Part III: I/O System

    Chapter 6. I/O System Overview

    Chapter 7. Devices

    Chapter 8. Local Filesystems

    Chapter 9. The Network Filesystem

    Chapter 10. Terminal Handling


Chapter 6. I/O System Overview


6.1. I/O Mapping from User to Device

Computers store and retrieve data through supporting peripheral I/O devices. These devices typically include mass-storage devices, such as disk drives, archival-storage devices, and network interfaces. Storage devices such as disks are accessed through I/O controllers that manage the operation of their attached devices according to I/O requests from the CPU.

Many hardware device peculiarities are hidden from the user by high-level kernel facilities, such as the filesystem and socket interfaces. Other such peculiarities are hidden from the bulk of the kernel itself by the I/O system. The I/O system consists of buffer-caching systems, general device-driver code, and drivers for specific hardware devices that must finally address peculiarities of the specific devices. An overview of the entire kernel is shown in Figure 6.1 (on page 216). The bottom third of the figure comprises the various I/O systems.

Figure 6.1. Kernel I/O structure.

There are three main kinds of I/O in FreeBSD: the character-device interface, the filesystem, and the socket interface with its related network devices. The character interface appears in the filesystem name space and provides unstructured access to the underlying hardware. The network devices do not appear in the filesystem; they are accessible through only the socket interface. Character devices are described in Section 6.2. The filesystem is described in Chapter 8. Sockets are described in Chapter 11.

A character-device interface comes in two styles that depend on the characteristics of the underlying hardware device. For some character-oriented hardware devices, such as terminal multiplexers, the interface is truly character oriented, although higher-level software, such as the terminal driver, may provide a line-oriented interface to applications. However, for block-oriented devices such as disks, a character-device interface is an unstructured or raw interface. For this interface, I/O operations do not go through the filesystem or the page cache; instead, they are made directly between the device and buffers in the application's virtual address space. Consequently, the size of the operations must be a multiple of the underlying block size required by the device, and, on some machines, the application's I/O buffer must be aligned on a suitable boundary.

Internal to the system, I/O devices are accessed through a set of entry points provided by each device's device driver. For a character-device interface, it accesses a cdevsw structure. A cdevsw structure is created for each device as the device is configured either at the time that the system is booted or later when the device is attached to the system.

Devices are identified by a device number that is constructed from a major and a minor device number. The major device number uniquely identifies the type of device (really of the device driver). Historically it was used as the index of the device's entry in the character-device table. FreeBSD 5.2 has no character-device table. As devices are configured, entries are created for the device in the /dev filesystem. Each entry in the /dev filesystem has a direct reference to its corresponding cdevsw entry. FreeBSD 5.2 assigns a unique major device number to each device when it is configured to provide compatibility for applications that look at it. But it is not used internally by the kernel or the device driver.

The minor device number is selected and interpreted solely by the device driver and is used by the driver to identify to which, of potentially many, hardware devices an I/O request refers. For disks, for example, minor device numbers identify a specific controller, disk drive, and partition. The minor device number may also specify a section of a device—for example, a channel of a multiplexed device, or optional handling parameters.

Device Drivers

A device driver is divided into three main sections:

  1. Autoconfiguration and initialization routines

  2. Routines for servicing I/O requests (the top half)

  3. Interrupt service routines (the bottom half)

The autoconfiguration portion of a driver is responsible for probing for a hardware device to see whether the latter is present and to initialize the device and any associated software state that is required by the device driver. This portion of the driver is typically called only once, either when the system is initialized or for transient devices when they are connected to the system. Autoconfiguration is described in Section 14.4.

The section of a driver that services I/O requests is invoked because of system calls or by the virtual-memory system. This portion of the device driver executes synchronously in the top half of the kernel and is permitted to block by calling the sleep() routine. We commonly refer to this body of code as the top half of a device driver.

Interrupt service routines are invoked when the system fields an interrupt from a device. Consequently, these routines cannot depend on any per-process state. Historically they did not have a thread context of their own, so they could not block. In FreeBSD 5.2 an interrupt has its own thread context, so it can block if it needs to do so. However, the cost of extra thread switches is sufficiently high that for good performance device drivers should attempt to avoid blocking. We commonly refer to a device driver's interrupt service routines as the bottom half of a device driver.

In addition to these three sections of a device driver, an optional crash-dump routine may be provided. This routine, if present, is invoked when the system recognizes an unrecoverable error and wishes to record the contents of physical memory for use in postmortem analysis. Most device drivers for disk controllers provide a crash-dump routine. The use of the crash-dump routine is described in Section 14.6.

I/O Queueing

Device drivers typically manage one or more queues of I/O requests in their normal operation. When an input or output request is received by the top half of the driver, it is recorded in a data structure that is placed on a per-device queue for processing. When an input or output operation completes, the device driver receives an interrupt from the controller. The interrupt service routine removes the appropriate request from the device's queue, notifies the requester that the command has completed, and then starts the next request from the queue. The I/O queues are the primary means of communication between the top and bottom halves of a device driver.

Because I/O queues are shared among asynchronous routines, access to the queues must be synchronized. Routines in both the top and bottom half of the device driver must acquire the mutex associated with the queue before manipulating it to avoid corruption from simultaneous modifications (mutexes were described in Section 4.3). For example, a bottom half interrupt might try to remove an entry that had not yet been fully linked in by the top half. Synchronization among multiple processes starting I/O requests is also serialized through the mutex associated with the queue.

Interrupt Handling

Interrupts are generated by devices to signal that an operation has completed or that a change in status has occurred. On receiving a device interrupt, the system schedules the appropriate device-driver interrupt service routine with one or more parameters that identify uniquely the device that requires service. These parameters are needed because device drivers typically support multiple devices of the same type. If the interrupting device's identity were not supplied with each interrupt, the driver would be forced to poll all the potential devices to identify the device that interrupted.

The system arranges for the unit-number parameter to be passed to the interrupt service routine for each device by installing the address of an auxiliary glue routine in the interrupt-vector table. This glue routine, rather than the actual interrupt service routine, is invoked to service the interrupt; it takes the following actions:

  1. Collects the relevant hardware parameters and places them in the space reserved for them by the device.

  2. Updates statistics on device interrupts.

  3. Schedules the interrupt service thread for the device.

  4. Clears the interrupt-pending flag in the hardware.

  5. Returns from the interrupt.

Because a glue routine is interposed between the interrupt-vector table and the interrupt service routine, special-purpose instructions that cannot be generated from C, which are needed by the hardware to support interrupts, can be kept out of the device driver. This interposition of a glue routine permits device drivers to be written without assembly language.


6.2. Character Devices

Almost all peripherals on the system, except network interfaces, have a character-device interface. A character device usually maps the hardware interface into a byte stream, similar to that of the filesystem. Character devices of this type include terminals (e.g., /dev/tty00), line printers (e.g, /dev/lp0), an interface to physical main memory (/dev/mem), and a bottomless sink for data and an endless source of end-of-file markers (/dev/null). Some of these character devices, such as terminal devices, may display special behavior on line boundaries but in general are still treated as byte streams.

Devices emulating terminals use buffers that are smaller than those used for disks. This buffering system involves small (usually 64-byte) blocks of characters kept in linked lists. Although all free character buffers are kept in a single free list, most device drivers that use them limit the number of characters that can be queued at one time for a single terminal port.

Devices such as high-speed graphics interfaces may have their own buffers or may always do I/O directly into the address space of the user; they too are classed as character devices. Some of these drivers may recognize special types of records and thus be further from the plain byte-stream model.

The character interface for disks is also called the raw device interface; it provides an unstructured interface to the device. Its primary task is to arrange for direct I/O to and from the device. The disk driver handles the asynchronous nature of I/O by maintaining and ordering an active queue of pending transfers. Each entry in the queue specifies whether it is for reading or writing, the main-memory address for the transfer, the device address for the transfer (usually a disk sector number), and the transfer size (in bytes).

All other restrictions of the underlying hardware are passed through the character interface to its clients, making character-device interfaces the furthest from the byte-stream model. Thus, the user process must abide by the sectoring restrictions imposed by the underlying hardware. For magnetic disks, the file offset and transfer size must be a multiple of the sector size. The character interface does not copy the user data into a kernel buffer before putting them on an I/O queue. Instead, it arranges to have the I/O done directly to or from the address space of the process. The size and alignment of the transfer is limited by the physical device. However, the transfer size is not restricted by the maximum size of the internal buffers of the system because these buffers are not used.

The character interface is typically used by only those system utility programs that have an intimate knowledge of the data structures on the disk. The character interface also allows user-level prototyping; for example, the 4.2BSD filesystem implementation was written and largely tested as a user process that used a raw disk interface, before the code was moved into the kernel.

Character devices are described by entries in the cdevsw structure. The entry points in this structure (see Table 6.1 on page 220) are used to support raw access to block-oriented devices such as disk, as well as normal access to character-oriented devices through the terminal driver. Raw devices support a subset of the entry points that correspond to those entry points found in block-oriented devices. The base set of entry points for all device drivers is described in this section; the additional set of entry points for block-oriented devices is given in Section 6.3.

Table 6.1. Entry points for character and raw device drivers.

Entry point

Function

open()

open the device

close()

close the device

read()

do an input operation

write()

do an output operation

ioctl()

do an I/O control operation

poll()

poll device for I/O readiness

stop()

stop output on the device

mmap()

map device offset to memory location

reset()

reinitialize device after a bus reset

Raw Devices and Physical I/O

Most raw devices differ from filesystems only in the way that they do I/O. Whereas filesystems read and write data to and from kernel buffers, raw devices transfer data to and from user buffers. Bypassing kernel buffers eliminates the memory-to-memory copy that must be done by filesystems but also denies applications the benefits of data caching. In addition, for devices that support both raw- and filesystem access, applications must take care to preserve consistency between data in the kernel buffers and data written directly to the device. The raw device should be used only when the filesystem is unmounted or mounted readonly. Raw-device access is used by many filesystem utilities, such as the filesystem check program, fsck, and by programs that read and write backup media—for example, tar, dump, and restore.

Because raw devices bypass kernel buffers, they are responsible for managing their own buffer structures. Most devices borrow swap buffers to describe their I/O. The read and write routines use the physio() routine to start a raw I/O operation (see Figure 6.2). The strategy parameter identifies a block-device strategy routine that starts I/O operations on the device. The buffer is used by physio() in constructing the request(s) made to the strategy routine. The device, read-write flag, and uio parameters completely specify the I/O operation that should be done. The maximum transfer size for the device is checked by physio () to adjust the size of each I/O transfer before the latter is passed to the strategy routine. This check allows the transfer to be done in sections according to the maximum transfer size supported by the device.

Figure 6.2. Algorithm for physical I/O.
void physio(
    device dev,
    struct uio *uio,
    int ioflag);
{
    allocate a swap buffer;
    while (uio is not exhausted) {
        mark the buffer busy for physical I/O;
        set up the buffer for a maximum-sized transfer;
        use device maximum I/O size to bound
            the transfer size;
        check user read/write access at uio location;
        lock the part of the user address space
            involved in the transfer into RAM;
        map the user pages into the buffer;
        call dev->strategy() to start the transfer;
        wait for the transfer to complete;
        unmap the user pages from the buffer;
        unlock the part of the address space previously
            locked;
        deduct the transfer size from the total number
            of data to transfer;
    }
    free swap buffer;
}

Raw-device I/O operations request the hardware device to transfer data directly to or from the data buffer in the user program's address space described by the uio parameter. Thus, unlike I/O operations that do direct memory access (DMA) from buffers in the kernel address space, raw I/O operations must check that the user's buffer is accessible by the device and must lock it into memory for the duration of the transfer.

Character-oriented Devices

Character-oriented I/O devices are typified by terminal ports, although they also include printers and other character- or line-oriented devices. These devices are usually accessed through the terminal driver, described in Chapter 10. The close tie to the terminal driver has heavily influenced the structure of character device drivers. For example, several entry points in the cdevsw structure exist for communication between the generic terminal handler and the terminal multiplexer hardware drivers.

Entry Points for Character Device Drivers

A device driver for a character device is defined by its entries in a cdevsw structure.

open | Open the device in preparation for I/O operations. A device's open entry point will be called for each open system call on a special-device file, or, internally, when a device is prepared for mounting a filesystem with the mount system call. The open() routine will commonly verify the integrity of the associated medium. For example, it will verify that the device was identified during the autoconfiguration phase and, for disk drives, that a medium is present and ready to accept commands.
close | Close a device. The close() routine is called after the final client interested in using the device terminates. These semantics are defined by the higher-level I/O facilities. Disk devices have nothing to do when a device is closed and thus use a null close() routine. Devices that support access to only a single client must mark the device as available once again.
read | Read data from a device. For raw devices, this entry point normally just calls the physio () routine with device-specific parameters. For terminal-oriented devices, a read request is passed immediately to the terminal driver. For other devices, a read request requires that the specified data be copied into the kernel's address space, typically with the uiomove() routine (see the end of Section 6.4), and then be passed to the device.
write | Write data to a device. This entry point is a direct parallel of the read entry point: Raw devices use physio (), terminal-oriented devices call the terminal driver to do this operation, and other devices handle the request internally.
ioctl | Do an operation other than a read or write. This entry point originally provided a mechanism to get and set device parameters for terminal devices; its use has expanded to other types of devices as well. Historically, ioctl() operations have varied widely from device to device.
poll | Check the device to see whether data are available for reading, or space is available for writing, data. The poll entry point is used by the select and poll system calls in checking file descriptors associated with device special files. For raw devices, a poll operation is meaningless, since data are not buffered. Here, the entry point is set to seltrue(), a routine that returns true for any poll request. For devices used with the terminal driver, this entry point is set to ttselect(), a routine described in Chapter 10.
mmap | Map a device offset into a memory address. This entry point is called by the virtual-memory system to convert a logical mapping to a physical address. For example, it converts an offset in /dev/mem to a kernel address.
kqfilter | Add the device to the kernel event list for the calling thread.


6.3. Disk Devices

Disk devices fill a central role in the UNIX kernel and thus have additional features and capabilities beyond those of the typical character device driver. Historically, UNIX provided two interfaces to disks. The first was a character-device interface that provided direct access to the disk in its raw form. This interface is still available in FreeBSD 5.2 and is described in Section 6.2. The second was a block-device interface that converted from the user abstraction of a disk as an array of bytes to the structure imposed by the underlying physical medium. Block devices were accessible directly through appropriate device special files. Block devices were eliminated in FreeBSD 5.2 because they were not needed by any common applications and added considerable complexity to the kernel.

Entry Points for Disk Device Drivers

Device drivers for disk devices contain all the usual character device entry points described in Section 6.2. In addition to those entry points there are three entry points that are used only for disk devices:

strategy | Start a read or write operation, and return immediately. I/O requests to or from filesystems located on a device are translated by the system into calls to the block I/O routines bread() and bwrite(). These block I/O routines in turn call the device's strategy routine to read or write data not in the memory cache. Each call to the strategy routine specifies a pointer to a buf structure containing the parameters for an I/O request. If the request is synchronous, the caller must sleep (on the address of the buf structure) until I/O completes.
dump | Write all physical memory to the device. The dump entry point saves the contents of memory on secondary storage. The system automatically takes a dump when it detects an unrecoverable error and is about to crash. The dump is used in postmortem analysis of the problem that caused the system to crash. The dump routine is invoked with context switching and interrupts disabled; thus, the device driver must poll for device status rather than wait for interrupts. At least one disk device is expected to support this entry point.

Sorting of Disk I/O Requests

The kernel provides a generic disksort() routine that can be used by all the disk device drivers to sort I/O requests into a drive's request queue using an elevator sorting algorithm. This algorithm sorts requests in a cyclic, ascending, block order, so that requests can be serviced with minimal one-way scans over the drive. This ordering was originally designed to support the normal read-ahead requested by the filesystem and also to counteract the filesystem's random placement of data on a drive. With the improved placement algorithms in the current filesystem, the effect of the disksort() routine is less noticeable; disksort() produces the largest effect when there are multiple simultaneous users of a drive.

The disksort() algorithm is shown in Figure 6.3. A drive's request queue is made up of two lists of requests ordered by block number. The first is the active list; the second is the next-pass list. The request at the front of the active list shows the current position of the drive. If next-pass list is not empty, it is made up of requests that lie before the current position. Each new request is sorted into either the active or the next-pass list, according to the request's location. When the heads reach the end of the active list, the next-pass list becomes the active list, an empty next-pass list is created, and the drive begins servicing the new active list.

Figure 6.3. Algorithm for disksort().
void disksort(
    drive queue *dq,
    buffer *bp);
{
    if (active list is empty) {
        place the buffer at the front of the active list;
        return;
    }
    if (request lies before the first active request) {
        locate the beginning of the next-pass list;
        sort by into the next-pass list;
    } else
        sort by into the active list;
}

Disk sorting can also be important on machines that have a fast processor but that do not sort requests within the device driver. Here, if a write of several Mbyte is honored in order of queueing, it can block other processes from accessing the disk while it completes. Sorting requests provides some scheduling, which more fairly distributes accesses to the disk controller.

Most modern disk controllers accept several concurrent I/O requests. The controller then sorts these requests to minimize the time needed to service them. If the controller could always manage all outstanding I/O requests, then there would be no need to have the kernel do any sorting. However, most controllers can handle only about 15 outstanding requests. Since a busy system can easily generate bursts of activity that exceed the number that the disk controller can manage simultaneously, disk sorting by the kernel is still necessary.

Disk Labels

A disk may be broken up into several partitions, each of which may be used for a separate filesystem or swap area. A disk label contains information about the partition layout and usage including type of filesystem, swap partition, or unused. For the fast filesystem, the partition usage contains enough additional information to enable the filesystem check program (fsck) to locate the alternate superblocks for the filesystem. The disk label also contains any other driver-specific information.

Having labels on each disk means that partition information can be different for each disk and that it carries over when the disk is moved from one system to another. It also means that, when previously unknown types of disks are connected to the system, the system administrator can use them without changing the disk driver, recompiling, and rebooting the system.

The label is located near the beginning of each drive—usually, in block zero. It must be located near the beginning of the disk to enable it to be used in the first-level bootstrap. Most architectures have hardware (or first-level) bootstrap code stored in read-only memory (ROM). When the machine is powered up or the reset button is pressed, the CPU executes the hardware bootstrap code from the ROM. The hardware bootstrap code typically reads the first few sectors on the disk into the main memory, then branches to the address of the first location that it read. The program stored in these first few sectors is the second-level bootstrap. Having the disk label stored in the part of the disk read as part of the hardware bootstrap allows the second-level bootstrap to have the disk-label information. This information gives it the ability to find the root filesystem and hence the files, such as the kernel, needed to bring up FreeBSD. The size and location of the second-level bootstrap are dependent on the requirements of the hardware bootstrap code. Since there is no standard for disk-label formats and the hardware bootstrap code usually understands only the vendor label, it is usually necessary to support both the vendor and the FreeBSD disk labels. Here, the vendor label must be placed where the hardware bootstrap ROM code expects it; the FreeBSD label must be placed out of the way of the vendor label but within the area that is read in by the hardware bootstrap code so that it will be available to the second-level bootstrap.

For example, on the PC architecture, the BIOS expects sector 0 of the disk to contain boot code, a slice table, and a magic number. BIOS slices can be used to break the disk up into several pieces. The BIOS brings in sector 0 and verifies the magic number. The sector 0 boot code then searches the slice table to determine which slice is marked active. This boot code then brings in the operating-system specific bootstrap from the active slice and, if marked bootable, runs it. This operating-system specific bootstrap includes the disk-label described above and the code to interpret it.


6.4. Descriptor Management and Services

For user processes, all I/O is done through descriptors. The user interface to descriptors was described in Section 2.6. This section describes how the kernel manages descriptors and how it provides descriptor services, such as locking and polling.

System calls that refer to open files take a file descriptor as an argument to specify the file. The file descriptor is used by the kernel to index into the descriptor table for the current process (kept in the filedesc structure, a substructure of the process structure for the process) to locate a file entry, or file structure. The relations of these data structures are shown in Figure 6.4.

Figure 6.4. File-descriptor reference to a file entry.

[View full size image]

The file entry provides a file type and a pointer to an underlying object for the descriptor. There are six object types supported in FreeBSD:

  • For data files, the file entry points to a vnode structure that references a substructure containing the filesystem-specific information described in Chapters 8 and 9. The vnode layer is described in Section 6.5. Special files do not have data blocks allocated on the disk; they are handled by the special-device filesystem that calls appropriate drivers to handle I/O for them.

  • For access to interprocess communication including networking, the FreeBSD file entry may reference a socket.

  • For unnamed high-speed local communication, the file entry will reference a pipe. Earlier FreeBSD systems used sockets for local communication, but optimized support was added for pipes to improve their performance.

  • For named high-speed local communication, the file entry will reference a fifo. As with pipes, optimized support was added for fifo's to improve performance.

  • For systems that have cryptographic support in hardware, the descriptor may provide direct access to that hardware.

  • To provide notification of kernel events, the descriptor will reference a kqueue.

The virtual-memory system supports the mapping of files into a process's address space. Here, the file descriptor must reference a vnode that will be partially or completely mapped into the user's address space.

Open File Entries

The set of file entries is the focus of activity for file descriptors. They contain the information necessary to access the underlying objects and to maintain common information.

The file entry is an object-oriented data structure. Each entry contains a type and an array of function pointers that translate the generic operations on file descriptors into the specific actions associated with their type. The operations that must be implemented for each type are as follows:

  • Read from the descriptor

  • Write to the descriptor

  • Poll the descriptor

  • Do ioctl operations on the descriptor

  • Collect stat information for the descriptor

  • Check to see if there are any kqueue events pending for the descriptor

  • Close and possibly deallocate the object associated with the descriptor

Note that there is no open routine defined in the object table. FreeBSD treats descriptors in an object-oriented fashion only after they are created. This approach was taken because the various descriptor types have different characteristics. Generalizing the interface to handle all types of descriptors at open time would have complicated an otherwise simple interface. Vnode descriptors are created by the open system call; socket descriptors are created by the socket system call; fifo descriptors are created by the pipe system call.

Each file entry has a pointer to a data structure that contains information specific to the instance of the underlying object. The data structure is opaque to the routines that manipulate the file entry. A reference to the data structure is passed on each call to a function that implements a file operation. All state associated with an instance of an object must be stored in that instance's data structure; the underlying objects are not permitted to manipulate the file entry themselves.

The read and write system calls do not take an offset in the file as an argument. Instead, each read or write updates the current file offset in the file according to the number of bytes transferred. The offset determines the position in the file for the next read or write. The offset can be set directly by the lseek system call. Since more than one process may open the same file, and each such process needs its own offset for the file, the offset cannot be stored in the per-object data structure. Thus, each open system call allocates a new file entry, and the open file entry contains the offset.

Some semantics associated with all file descriptors are enforced at the descriptor level, before the underlying system call is invoked. These semantics are maintained in a set of flags associated with the descriptor. For example, the flags record whether the descriptor is open for reading, writing, or both reading and writing. If a descriptor is marked as open for reading only, an attempt to write it will be caught by the descriptor code. Thus, the functions defined for doing reading and writing do not need to check the validity of the request; we can implement them knowing that they will never receive an invalid request.

The application-visible flags are described in the next subsection. In addition to the application-visible flags, the flags field also has information on whether the descriptor holds a shared or exclusive lock on the underlying file. The locking primitives could be extended to work on sockets, as well as on files. However, the descriptors for a socket rarely refer to the same file entry. The only way for two processes to share the same socket descriptor is for a parent to share the descriptor with its child by forking or for one process to pass the descriptor to another in a message.

Each file entry has a reference count. A single process may have multiple references to the entry because of calls to the dup or fcntl system calls. Also, file structures are inherited by the child process after a fork, so several different processes may reference the same file entry. Thus, a read or write by either process on the twin descriptors will advance the file offset. This semantic allows two processes to read the same file or to interleave output to the same file. Another process that has independently opened the file will refer to that file through a different file structure with a different file offset. This functionality was the original reason for the existence of the file structure; the file structure provides a place for the file offset between the descriptor and the underlying object.

Each time that a new reference is created, the reference count is incremented. When a descriptor is closed (any one of (1) explicitly with a close, (2) implicitly after an exec because the descriptor has been marked as close-on-exec, or (3) on process exit), the reference count is decremented. When the reference count drops to zero, the file entry is freed.

The close-on-exec flag is kept in the descriptor table rather than in the file entry. This flag is not shared among all the references to the file entry because it is an attribute of the file descriptor itself. The close-on-exec flag is the only piece of information that is kept in the descriptor table rather than being shared in the file entry.

Management of Descriptors

The fcntl system call manipulates the file structure. It can be used to make the following changes to a descriptor:

  • Duplicate a descriptor as though by a dup system call.

  • Get or set the close-on-exec flag. When a process forks, all the parent's descriptors are duplicated in the child. The child process then execs a new process.

    Any of the child's descriptors that were marked close-on-exec are closed. The remaining descriptors are available to the newly executed process.

  • Set the no-delay (O_NONBLOCK) flag to put the descriptor into nonblocking mode. In nonblocking mode, if any data are available for a read operation, or if any space is available for a write operation, an immediate partial read or write is done. If no data are available for a read operation, or if a write operation would block, the system call returns an error (EAGAIN) showing that the operation would block, instead of putting the process to sleep. This facility is not implemented for local filesystems in FreeBSD, because local-filesystem I/O is always expected to complete within a few milliseconds.

  • Set the synchronous (O_FSYNC) flag to force all writes to the file to be written synchronously to the disk.

  • Set the direct (O_DIRECT) flag to request that the kernel attempt to write the data directly from the user application to the disk rather than copying it via kernel buffers.

  • Set the append (O_APPEND) flag to force all writes to append data to the end of the file, instead of at the descriptor's current location in the file. This feature is useful when, for example, multiple processes are writing to the same log file.

  • Set the asynchronous (O_ASYNC) flag to request that the kernel watch for a change in the status of the descriptor, and arrange to send a signal (SIGIO) when a read or write becomes possible.

  • Send a signal to a process when an exception condition arises, such as when urgent data arrive on an interprocess-communication channel.

  • Set or get the process identifier or process-group identifier to which the two I/O-related signals in the previous steps should be sent.

  • Test or change the status of a lock on a range of bytes within an underlying file. Locking operations are described later in this section.

The implementation of the dup system call is easy. If the process has reached its limit on open files, the kernel returns an error. Otherwise, the kernel scans the current process's descriptor table, starting at descriptor zero, until it finds an unused entry. The kernel allocates the entry to point to the same file entry as does the descriptor being duplicated. The kernel then increments the reference count on the file entry and returns the index of the allocated descriptor-table entry. The fcntl system call provides a similar function, except that it specifies a descriptor from which to start the scan.

Sometimes, a process wants to allocate a specific descriptor-table entry. Such a request is made with the dup2 system call. The process specifies the descriptor-table index into which the duplicated reference should be placed. The kernel implementation is the same as for dup, except that the scan to find a free entry is changed to close the requested entry if that entry is open and then to allocate the entry as before. No action is taken if the new and old descriptors are the same.

The system implements getting or setting the close-on-exec flag via the fcntl system call by making the appropriate change to the flags field of the associated descriptor-table entry. Other attributes that fcntl manipulates operate on the flags in the file entry. However, the implementation of the various flags cannot be handled by the generic code that manages the file entry. Instead, the file flags must be passed through the object interface to the type-specific routines to do the appropriate operation on the underlying object. For example, manipulation of the non-blocking flag for a socket must be done by the socket layer, since only that layer knows whether an operation can block.

The implementation of the ioctl system call is broken into two major levels. The upper level handles the system call itself. The ioctl call includes a descriptor, a command, and pointer to a data area. The command argument encodes what the size is of the data area for the parameters and whether the parameters are input, output, or both input and output. The upper level is responsible for decoding the command argument, allocating a buffer, and copying in any input data. If a return value is to be generated and there is no input, the buffer is zeroed. Finally, the ioctl is dispatched through the file-entry ioctl function, along with the I/O buffer, to the lower-level routine that implements the requested operation.

The lower level does the requested operation. Along with the command argument, it receives a pointer to the I/O buffer. The upper level has already checked for valid memory references, but the lower level may do more precise argument validation because it knows more about the expected nature of the arguments. However, it does not need to copy the arguments in or out of the user process. If the command is successful and produces output, the lower level places the results in the buffer provided by the top level. When the lower level returns, the upper level copies the results to the process.

Asynchronous I/O

Historically, UNIX systems did not have the ability to do asynchronous I/O beyond the ability to do background writes to the filesystem. An asynchronous I/O interface was defined by the POSIX.lb-1993 realtime group. Shortly after its ratification, an implementation was added to FreeBSD.

An asynchronous read is started with aio_read; an asynchronous write is started with aio_write. The kernel builds an asynchronous I/O request structure that contains all the information needed to do the requested operation. If the request cannot be immediately satisfied from kernel buffers, the request structure is queued for processing by an asynchronous kernel-based I/O daemon and the system call returns. The next available asynchronous I/O daemon handles the request using the usual kernel synchronous I/O path.

When the daemon finishes the I/O, the asynchronous I/O structure is marked as finished along with a return value or error code. The application uses the aio_error system call to poll to find if the I/O is complete. This call is implemented by checking the status of the asynchronous I/O request structure created by the kernel. If an application gets to the point where it cannot proceed until an I/O completes, it can use the aio_suspend system call to wait until an I/O is done. Here, the application is put to sleep on its asynchronous I/O request structure and is awakened by the asynchronous I/O daemon when the I/O completes. Alternatively, the application can request that a specified signal be sent when the I/O is done.

The aio_return system call gets the return value from the asynchronous request once aio_error, aio_suspend, or the arrival of a completion signal has indicated that the I/O is done. FreeBSD has also added the nonstandard aio_waitcomplete system call that combines the functions of aio_suspend and aio_return into a single operation. For either aio_return or aio_waitcomplete, the return information is copied out to the application from the asynchronous I/O request structure and the asynchronous I/O request structure is then freed.

File-Descriptor Locking

Early UNIX systems had no provision for locking files. Processes that needed to synchronize access to a file had to use a separate lock file. A process would try to create a lock file. If the creation succeeded, then the process could proceed with its update; if the creation failed, the process would wait and then try again. This mechanism had three drawbacks:

  1. Processes consumed CPU time by looping over attempts to create locks.

  2. Locks left lying around because of system crashes had to be removed (normally in a system-startup command script).

  3. Processes running as the special system-administrator user, the superuser, are always permitted to create files, and so were forced to use a different mechanism.

Although it is possible to work around all these problems, the solutions are not straightforward, so a mechanism for locking files was added in 4.2BSD.

The most general locking schemes allow multiple processes to update a file concurrently. Several of these techniques are discussed in Peterson [1983]. A simpler technique is to serialize access to a file with locks. For standard system applications, a mechanism that locks at the granularity of a file is sufficient. So, 4.2BSD and 4.3BSD provided only a fast, whole-file locking mechanism. The semantics of these locks include allowing locks to be inherited by child processes and releasing locks only on the last close of a file.

Certain applications require the ability to lock pieces of a file. Locking facilities that support a byte-level granularity are well understood. Unfortunately, they are not powerful enough to be used by database systems that require nested hierarchical locks, but are complex enough to require a large and cumbersome implementation compared to the simpler whole-file locks. Because byte-range locks are mandated by the POSIX standard, the developers added them to BSD reluctantly. The semantics of byte-range locks come from the initial implementation of locks in System V, which included releasing all locks held by a process on a file every time a close system call was done on a descriptor referencing that file. The 4.2BSD whole-file locks are removed only on the last close. A problem with the POSIX semantics is that an application can lock a file, then call a library routine that opens, reads, and closes the locked file. Calling the library routine will have the unexpected effect of releasing the locks held by the application. Another problem is that a file must be open for writing to be allowed to get an exclusive lock. A process that does not have permission to open a file for writing cannot get an exclusive lock on that file. To avoid these problems, yet remain POSIX compliant, FreeBSD provides separate interfaces for byte-range locks and whole-file locks. The byte-range locks follow the POSIX semantics; the whole-file locks follow the traditional 4.2BSD semantics. The two types of locks can be used concurrently; they will serialize against each other properly.

Both whole-file locks and byte-range locks use the same implementation; the whole-file locks are implemented as a range lock over an entire file. The kernel handles the other differing semantics between the two implementations by having the byte-range locks be applied to processes, whereas the whole-file locks are applied to descriptors. Because descriptors are shared with child processes, the whole-file locks are inherited. Because the child process gets its own process structure, the byte-range locks are not inherited. The last-close versus every-close semantics are a small bit of special-case code in the close routine that checks whether the underlying object is a process or a descriptor. It releases locks on every call if the lock is associated with a process and only when the reference count drops to zero if the lock is associated with a descriptor.

Locking schemes can be classified according to the extent that they are enforced. A scheme in which locks are enforced for every process without choice is said to use mandatory locks, whereas a scheme in which locks are enforced for only those processes that request them is said to use advisory locks. Clearly, advisory locks are effective only when all programs accessing a file use the locking scheme. With mandatory locks, there must be some override policy implemented in the kernel. With advisory locks, the policy is left to the user programs. In the FreeBSD system, programs with superuser privilege are allowed to override any protection scheme. Because many of the programs that need to use locks must also run as the superuser, 4.2BSD implemented advisory locks rather than creating an additional protection scheme that was inconsistent with the UNIX philosophy or that could not be used by privileged programs. The use of advisory locks carried over to the POSIX specification of byte-range locks and is retained in FreeBSD.

The FreeBSD file-locking facilities allow cooperating programs to apply advisory shared or exclusive locks on ranges of bytes within a file. Only one process may have an exclusive lock on a byte range, whereas multiple shared locks may be present. A shared and an exclusive lock cannot be present on a byte range at the same time. If any lock is requested when another process holds an exclusive lock, or an exclusive lock is requested when another process holds any lock, the lock request will block until the lock can be obtained. Because shared and exclusive locks are only advisory, even if a process has obtained a lock on a file, another process may access the file if it ignores the locking mechanism.

So that there are no races between creating and locking a file, a lock can be requested as part of opening a file. Once a process has opened a file, it can manipulate locks without needing to close and reopen the file. This feature is useful, for example, when a process wishes to apply a shared lock, to read information, to determine whether an update is required, and then to apply an exclusive lock and update the file.

A request for a lock will cause a process to block if the lock cannot be obtained immediately. In certain instances, this blocking is unsatisfactory. For example, a process that wants only to check whether a lock is present would require a separate mechanism to find out this information. Consequently, a process can specify that its locking request should return with an error if a lock cannot be obtained immediately. Being able to request a lock conditionally is useful to daemon processes that wish to service a spooling area. If the first instance of the daemon locks the directory where spooling takes place, later daemon processes can easily check to see whether an active daemon exists. Since locks exist only while the locking processes exist, locks can never be left active after the processes exit or if the system crashes.

The implementation of locks is done on a per-filesystem basis. The implementation for the local filesystems is described in Section 7.5. A network-based filesystem has to coordinate locks with a central lock manager that is usually located on the server exporting the filesystem. Client lock requests must be sent to the lock manager. The lock manager arbitrates among lock requests from processes running on its server and from the various clients to which it is exporting the filesystem. The most complex operation for the lock manager is recovering lock state when a client or server is rebooted or becomes partitioned from the rest of the network. The FreeBSD network-based lock manager is described in Section 9.3.

Multiplexing I/O on Descriptors

A process sometimes wants to handle I/O on more than one descriptor. For example, consider a remote login program that wants to read data from the keyboard and to send them through a socket to a remote machine. This program also wants to read data from the socket connected to the remote end and to write them to the screen. If a process makes a read request when there are no data available, it is normally blocked in the kernel until the data become available. In our example, blocking is unacceptable. If the process reads from the keyboard and blocks, it will be unable to read data from the remote end that are destined for the screen. The user does not know what to type until more data arrive from the remote end, so the session deadlocks. Conversely, if the process reads from the remote end when there are no data for the screen, it will block and will be unable to read from the terminal. Again, deadlock would occur if the remote end were waiting for output before sending any data. There is an analogous set of problems to blocking on the writes to the screen or to the remote end. If a user has stopped output to his screen by typing the stop character, the write will block until the user types the start character. In the meantime, the process cannot read from the keyboard to find out that the user wants to flush the output.

FreeBSD provides three mechanisms that permit multiplexing I/O on descriptors: polling I/O, nonblocking I/O, and signal-driven I/O. Polling is done with the select or poll system call, described in the next subsection. Operations on non-blocking descriptors finish immediately, partially complete an input or output operation and return a partial count, or return an error that shows that the operation could not be completed at all. Descriptors that have signaling enabled cause the associated process or process group to be notified when the I/O state of the descriptor changes.

There are four possible alternatives that avoid the blocking problem:

  1. Set all the descriptors into nonblocking mode. The process can then try operations on each descriptor in turn to find out which descriptors are ready to do I/O. The problem with this busy-waiting approach is that the process must run continuously to discover whether there is any I/O to be done, wasting CPU cycles.

  2. Enable all descriptors of interest to signal when I/O can be done. The process can then wait for a signal to discover when it is possible to do I/O. The drawback to this approach is that signals are expensive to catch. Hence, signal-driven I/O is impractical for applications that do moderate to large amounts of I/O.

  3. Have the system provide a method for asking which descriptors are capable of doing I/O. If none of the requested descriptors are ready, the system can put the process to sleep until a descriptor becomes ready. This approach avoids the problem of deadlock because the process will be awakened whenever it is possible to do I/O and will be told which descriptor is ready. The drawback is that the process must do two system calls per operation: one to poll for the descriptor that is ready to do I/O and another to do the operation itself.

  4. Have the process notify the system of all the descriptors that it is interested in reading and then do a blocking read on that set of descriptors. When the read returns, the process is notified on which descriptor the read completed. The benefit of this approach is that the process does a single system call to specify the set of descriptors, then loops doing only reads [Accetta et al., 1986; Lemon, 2001].

The first approach is available in FreeBSD as nonblocking I/O. It typically is used for output descriptors because the operation typically will not block. Rather than doing a select or poll, which nearly always succeeds, followed immediately by a write, it is more efficient to try the write and revert to using select or poll only during periods when the write returns a blocking error. The second approach is available in FreeBSD as signal-driven I/O. It typically is used for rare events, such as the arrival of out-of-band data on a socket. For such rare events, the cost of handling an occasional signal is lower than that of checking constantly with select or poll to find out whether there are any pending data.

The third approach is available in FreeBSD via the select or poll system call. Although less efficient than the fourth approach, it is a more general interface. In addition to handling reading from multiple descriptors, it handles writes to multiple descriptors, notification of exceptional conditions, and timeout when no I/O is possible.

The select and poll interfaces provide the same information. They differ only in their programming interface. The select interface was first developed in 4.2BSD with the introduction of socket-based interprocess communication. The poll interface was introduced in System V several years later with their competing STREAMS-based interprocess communication. Although STREAMS has fallen into disuse, the poll interface has proven popular enough to be retained. The FreeBSD kernel supports both interfaces.

The select system call is of the form

int error = select(
    int numfds,
    fd_set *readfds,
    fd_set *writefds,
    fd_set *exceptfds,
    struct timeval *timeout);

It takes three masks of descriptors to be monitored, corresponding to interest in reading, writing, and exceptional conditions. In addition, it takes a timeout value for returning from select if none of the requested descriptors becomes ready before a specified amount of time has elapsed. The select call returns the same three masks of descriptors after modifying them to show the descriptors that are able to do reading, to do writing, or that have an exceptional condition. If none of the descriptors has become ready in the timeout interval, select returns showing that no descriptors are ready for I/O. If a timeout value is given and a descriptor becomes ready before the specified timeout period, the time that select spends waiting for I/O to become ready is subtracted from the time given.

The poll interface copies in an array of pollfd structures, one array entry for each descriptor of interest. The pollfd structure contains three elements:

  • The file descriptor to poll

  • A set of flags describing the information being sought

  • A set of flags set by the kernel showing the information that was found

The flags specify availability of normal or out-of-band data for reading and the availability of buffer space for normal or out-of-band data writing. The return flags can also specify that an error has occurred on the descriptor, that the descriptor has been disconnected, or that the descriptor is not open. These error conditions are raised by the select call by indicating that the descriptor with the error is ready to do I/O. When the application attempts to do the I/O, the error is returned by the read or write system call. Like the select call, the poll call takes a timeout value to specify the maximum time to wait. If none of the requested descriptors becomes ready before the specified amount of time has elapsed, the poll call returns. If a timeout value is given and a descriptor becomes ready before the specified timeout period, the time that poll spends waiting for I/O to become ready is subtracted from the time given.

Implementation of Select

The implementation of select is divided into a generic top layer and many deviceor socket-specific bottom pieces. At the top level, select or poll decodes the user's request and then calls the appropriate lower-level poll functions. The select and poll system calls have different top layers to determine the sets of descriptors to be polled but use all the same device- and socket-specific bottom pieces. Only the select top layer will be described here. The poll top layer is implemented in a completely analogous way.

The select top level takes the following steps:

1. Copy and validate the descriptor masks for read, write, and exceptional conditions. Doing validation requires checking that each requested descriptor is currently open by the process.

2. Set the selecting flag for the thread.

3. For each descriptor in each mask, call the device's poll routine. If the descriptor is not able to do the requested I/O operation, the device poll routine is responsible for recording that the thread wants to do I/O. When I/O becomes possible for the descriptor—usually as a result of an interrupt from the underlying device—a notification must be issued for the selecting thread.

4. Because the selection process may take a long time, the kernel does not want to block out I/O during the time it takes to poll all the requested descriptors. Instead, the kernel arranges to detect the occurrence of I/O that may affect the status of the descriptors being polled. When such I/O occurs, the select-notification routine, selwakeup(), clears the selecting flag. If the top-level select code finds that the selecting flag for the thread has been cleared while it has been doing the polling, and it has not found any descriptors that are ready to do an operation, then the top level knows that the polling results are incomplete and must be repeated starting at step 2. The other condition that requires the polling to be repeated is a collision. Collisions arise when multiple threads attempt to poll the same descriptor at the same time. Because the poll routines have only enough space to record a single thread identifier, they cannot track multiple threads that need to be awakened when I/O is possible. In such rare instances, all threads that are selecting must be awakened.

5. If no descriptors are ready and the select specified a timeout, the kernel posts a timeout for the requested amount of time. The thread goes to sleep, giving the address of the kernel global variable selwait. Normally, a descriptor will become ready and the thread will be notified by selwakeup(). When the thread is awakened, it repeats the polling process and returns the available descriptors. If none of the descriptors become ready before the timer expires, the thread returns with a timed-out error and an empty list of available descriptors. If a timeout value is given and a descriptor becomes ready before the specified timeout period, the time that select spent waiting for I/O to become ready is subtracted from the time given.

Each of the low-level polling routines in the terminal drivers and the network protocols follows roughly the same set of steps. A piece of the poll routine for a terminal driver is shown in Figure 6.5 (on page 238). The steps involved in a device poll routine are as follows:

1. The socket or device poll entry is called with one or more of the flags POLLIN, POLLOUT, or POLLRDBAND (exceptional condition). The example in Figure 6.4 shows the POLLIN case; the other cases are similar.

2. The poll routine sets flags showing the operations that are possible. In Figure 6.5, it is possible to read a character if the number of unread characters is greater than zero. In addition, if the carrier has dropped, it is possible to get a read error. A return from select does not necessarily mean that there are data to read; rather, it means that a read will not block.

3. If the requested operation is not possible, the thread identifier is recorded with the socket or device for later notification. In Figure 6.5, the recording is done by the selrecord() routine. That routine first checks to see whether any thread is recorded for this socket or device. If none is recorded, then the current thread is recorded and the selinfo structure is linked onto the list of events associated with this thread. The second if statement checks for a collision. If the thread recorded with the structure is not ourselves, then a collision has occurred.

4. If multiple threads are selecting on the same socket or device, a collision is recorded for the socket or device, because the structure has only enough space for a single thread identifier. In Figure 6.5, a collision occurs when the second if statement in the selrecord() function is true. There is a tty structure containing a selinfo structure for each terminal line (or pseudo-terminal) on the machine. Normally, only one thread at a time is selecting to read from the terminal, so collisions are rare.

Selecting threads must be notified when I/O becomes possible. The steps involved in a status change awakening a thread are as follows:

1. The device or socket detects a change in status. Status changes normally occur because of an interrupt (e.g., a character coming in from a keyboard or a packet arriving from the network).

Figure 6.5. Select code from the pseudo-terminal driver to check for data to read.
struct selinfo {
    TAILQ_ENTRY(selinfo) si_thrlist;/* list hung off thread */
    struct  thread *si_thread;   /* thread that is waiting */
    short   si_flags;        /* SI_COLL - collision occurred */
};

struct tty *tp;

    if (events & POLLIN) {
        if (nread > 0 || (tp->t_state & TS_CARR_ON) == 0)
            revents | = POLLIN;
        else
            selrecord(curthread, &tp->t_rsel);
    }

void selrecord(
    struct thread *selector,
    struct selinfo *sip);
{

    if (sip->si_thread == NULL) {
        sip->si_thread = selector;
        TAILQ_INSERT_TAIL(&selector->td_selq, sip, si_thrlist);
    } else if (sip->si_thread != selector) {
        sip->si_flags | = SI_COLL;
    }
}

2. Selwakeup() is called with a pointer to the selinfo structure used by selrecord() to record the thread identifier and with a flag showing whether a collision occurred.

3. If the thread is sleeping on selwait, it is made runnable (or is marked ready, if it is stopped). If the thread is sleeping on some event other than selwait, it is not made runnable. A spurious call to selwakeup() can occur when the thread returns from select to begin processing one descriptor and then another descriptor on which it had been selecting also becomes ready.

4. If the thread has its selecting flag set, the flag is cleared so that the kernel will know that its polling results are invalid and must be recomputed.

5. If a collision has occurred, all sleepers on selwait are awakened to rescan to see whether one of their descriptors became ready. Awakening all selecting threads is necessary because the selrecord() routine could not record all the threads that needed to be awakened. Hence, it has to wake up all threads that could possibly have been interested. Empirically, collisions occur infrequently. If they were a frequent occurrence, it would be worthwhile to store multiple thread identifiers in the selinfo structure.

Movement of Data Inside the Kernel

Within the kernel, I/O data are described by an array of vectors. Each I/O vector or iovec has a base address and a length. The I/O vectors are identical to the I/O vectors used by the readv and writev system calls.

The kernel maintains another structure, called a uio structure, that holds additional information about the I/O operation. A sample uio structure is shown in Figure 6.6; it contains the following:

  • A pointer to the iovec array

  • The number of elements in the iovec array

  • The file offset at which the operation should start

  • The sum of the lengths of the I/O vectors

  • A flag showing whether the source and destination are both within the kernel or whether the source and destination are split between the user and the kernel

  • A flag showing whether the data are being copied from the uio structure to the kernel (UIO_WRITE) or from the kernel to the uio structure (UIO_READ)

  • A pointer to the thread whose data area is described by the uio structure (the pointer is NULL if the uio structure describes an area within the kernel)

Figure 6.6. A uio structure.

All I/O within the kernel is described with iovec and uio structures. System calls such as read and write that are not passed an iovec create a uio to describe their arguments; this uio structure is passed to the lower levels of the kernel to specify the parameters of an I/O operation. Eventually, the uio structure reaches the part of the kernel responsible for moving the data to or from the process address space: the filesystem, the network, or a device driver. In general, these parts of the kernel do not interpret uio structures directly. Instead, they arrange a kernel buffer to hold the data and then use uiomove() to copy the data to or from the buffer or buffers described by the uio structure. The uiomove() routine is called with a pointer to a kernel data area, a data count, and a uio structure. As it moves data, it updates the counters and pointers of the iovec and uio structures by a corresponding amount. If the kernel buffer is not as large as the areas described by the uio structure, the uio structure will point to the part of the process address space just beyond the location completed most recently. Thus, while servicing a request, the kernel may call uiomove() multiple times, each time giving a pointer to a new kernel buffer for the next block of data.

Character device drivers that do not copy data from the process generally do not interpret the uio structure. Instead, there is one low-level kernel routine that arranges a direct transfer to or from the address space of the process. Here, a separate I/O operation is done for each iovec element, calling back to the driver with one piece at a time.


6.5. The Virtual-Filesystem Interface

In early UNIX systems, the file entries directly referenced the local filesystem inode. An inode is a data structure that describes the contents of a file; it is more fully described in Section 8.2. This approach worked fine when there was a single filesystem implementation. However, with the advent of multiple filesystem types, the architecture had to be generalized. The new architecture had to support importing of filesystems from other machines including other machines that were running different operating systems.

One alternative would have been to connect the multiple filesystems into the system as different file types. However, this approach would have required massive restructuring of the internal workings of the system, because current directories, references to executables, and several other interfaces used inodes instead of file entries as their point of reference. Thus, it was easier and more logical to add a new object-oriented layer to the system below the file entry and above the inode. This new layer was first implemented by Sun Microsystems, which called it the virtual-node, or vnode, layer. Interfaces in the system that had referred previously to inodes were changed to reference generic vnodes. A vnode used by a local filesystem would refer to an inode. A vnode used by a remote filesystem would refer to a protocol control block that described the location and naming information necessary to access the remote file.

Contents of a Vnode

The vnode is an extensible object-oriented interface. It contains information that is generically useful independent of the underlying filesystem object that it represents. The information stored in a vnode includes the following:

  • Flags are used for identifying generic attributes. An example generic attribute is a flag to show that a vnode represents an object that is the root of a filesystem.

  • The various reference counts include the number of file entries that are open for reading and/or writing that reference the vnode, the number of file entries that are open for writing that reference the vnode, and the number of pages and buffers that are associated with the vnode.

  • A pointer to the mount structure describes the filesystem that contains the object represented by the vnode.

  • Various information to do file read-ahead.

  • A reference to the vm_object associated with the vnode.

  • A reference to state about special devices, sockets, and fifos.

  • A mutex to protect the flags and counters within the vnode.

  • A lock-manager lock to protect parts of the vnode that may change while it has an I/O operation in progress.

  • Fields used by the name cache to track the names associated with the vnode.

  • A pointer to the set of vnode operations defined for the object. These operations are described in the next subsection.

  • A pointer to private information needed for the underlying object. For the local filesystem, this pointer will reference an inode; for NFS, it will reference an nfsnode.

  • The type of the underlying object (e.g., regular file, directory, character device, etc.) is given. The type information is not strictly necessary, since a vnode client could always call a vnode operation to get the type of the underlying object. However, because the type often is needed, the type of underlying objects does not change, and it takes time to call through the vnode interface, the object type is cached in the vnode.

  • There are clean and dirty buffers associated with the vnode. Each valid buffer in the system is identified by its associated vnode and the starting offset of its data within the object that the vnode represents. All the buffers that have been modified but have not yet been written back are stored on their vnode dirty-buffer list. All buffers that have not been modified or have been written back since they were last modified are stored on their vnode clean list. Having all the dirty buffers for a vnode grouped onto a single list makes the cost of doing an fsync system call to flush all the dirty blocks associated with a file proportional to the amount of dirty data. In some UNIX systems, the cost is proportional to the smaller of the size of the file or the size of the buffer pool. The list of clean buffers is used to free buffers when a file is deleted. Since the file will never be read again, the kernel can immediately cancel any pending I/O on its dirty buffers and reclaim all its clean and dirty buffers and place them at the head of the buffer free list, ready for immediate reuse.

  • A count is kept of the number of buffer write operations in progress. To speed the flushing of dirty data, the kernel does this operation by doing asynchronous writes on all the dirty buffers at once. For local filesystems, this simultaneous push causes all the buffers to be put into the disk queue so that they can be sorted into an optimal order to minimize seeking. For remote filesystems, this simultaneous push causes all the data to be presented to the network at once so that it can maximize their throughput. System calls that cannot return until the data are on stable store (such as fsync) can sleep on the count of pending output operations, waiting for the count to reach zero.

The position of vnodes within the system was shown in Figure 6.1. The vnode itself is connected into several other structures within the kernel, as shown in Figure 6.7. Each mounted filesystem within the kernel is represented by a generic mount structure that includes a pointer to a filesystem-specific control block. All the vnodes associated with a specific mount point are linked together on a list headed by this generic mount structure. Thus, when it is doing a sync system call for a filesystem, the kernel can traverse this list to visit all the files active within that filesystem. Also shown in the figure are the lists of clean and dirty buffers associated with each vnode. Finally, there is a free list that links together all the vnodes in the system that are inactive (not currently referenced). The free list is used when a filesystem needs to allocate a new vnode so that the latter can open a new file; see Section 6.4.

Figure 6.7. Vnode linkages. Key: D—dirty buffer; C—clean buffer.

Vnode Operations

Vnodes are designed as an object-oriented interface. Thus, the kernel manipulates them by passing requests to the underlying object through a set of defined operations. Because of the many varied filesystems that are supported in FreeBSD, the set of operations defined for vnodes is both large and extensible. Unlike the original Sun Microsystems vnode implementation, the one in FreeBSD allows dynamic addition of vnode operations either at system boot time or when a new filesystem is dynamically loaded into the kernel. As part of activating a filesystem, it registers the set of vnode operations that it is able to support. The kernel then builds a table that lists the union of all operations supported by any filesystem. From that table, it builds an operations vector for each filesystem. Supported operations are filled in with the entry point registered by the filesystem. Filesystems may opt to have unsupported operations filled in with either a default routine (typically a routine to bypass the operation to the next lower layer; see Section 6.7) or a routine that returns the characteristic error "operation not supported" [Heidemann & Popek, 1994].

In 4.3BSD, the local filesystem code provided both the semantics of the hierarchical filesystem naming and the details of the on-disk storage management. These functions are only loosely related. To enable experimentation with other disk-storage techniques without having to reproduce the entire naming semantics, 4.4BSD split the naming and storage code into separate modules. The vnode-level operations define a set of hierarchical filesystem operations. Below the naming layer are a separate set of operations defined for storage of variable-sized objects using a flat name space. About 60 percent of the traditional filesystem code became the name-space management, and the remaining 40 percent became the code implementing the on-disk file storage. The 4.4BSD system used this division to support two distinct disk layouts: the traditional fast filesystem and a log-structured filesystem. Support for the log-structured filesystem was dropped in FreeBSD due to lack of anyone willing to maintain it but remains as a primary filesystem in NetBSD. The naming and disk-storage scheme are described in Chapter 8.

Pathname Translation

The translation of a pathname requires a series of interactions between the vnode interface and the underlying filesystems. The pathname-translation process proceeds as follows:

1. The pathname to be translated is copied in from the user process or, for a remote filesystem request, is extracted from the network buffer.

2. The starting point of the pathname is determined as either the root directory or the current directory (see Section 2.7). The vnode for the appropriate directory becomes the lookup directory used in the next step.

3. The vnode layer calls the filesystem-specific lookup() operation and passes to that operation the remaining components of the pathname and the current lookup directory. Typically, the underlying filesystem will search the lookup directory for the next component of the pathname and will return the resulting vnode (or an error if the name does not exist).

4. If an error is returned, the top level returns the error. If the pathname has been exhausted, the pathname lookup is done, and the returned vnode is the result of the lookup. If the pathname has not been exhausted, and the returned vnode is not a directory, then the vnode layer returns the "not a directory" error. If there are no errors, the top layer checks to see whether the returned directory is a mount point for another filesystem. If it is, then the lookup directory becomes the mounted filesystem; otherwise, the lookup directory becomes the vnode returned by the lower layer. The lookup then iterates with step 3.

Although it may seem inefficient to call through the vnode interface for each pathname component, doing so usually is necessary. The reason is that the underlying filesystem does not know which directories are being used as mount points. Since a mount point will redirect the lookup to a new filesystem, it is important that the current filesystem not proceed past a mounted directory. Although it might be possible for a local filesystem to be knowledgeable about which directories are mount points, it is nearly impossible for a server to know which of the directories within its exported filesystems are being used as mount points by its clients. Consequently, the conservative approach of traversing only a single pathname component per lookup() call is used. There are a few instances where a filesystem will know that there are no further mount points in the remaining path, and will traverse the rest of the pathname. An example is crossing into a portal, described in Section 6.7.

Exported Filesystem Services

The vnode interface has a set of services that the kernel exports from all the filesystems supported under the interface. The first of these is the ability to support the update of generic mount options. These options include the following:

noexec | Do not execute any files on the filesystem. This option is often used when a server exports binaries for a different architecture that cannot be executed on the server itself. The kernel will even refuse to execute shell scripts; if a shell script is to be run, its interpreter must be invoked explicitly.
nosuid | Do not honor the set-user-id or set-group-id flags for any executables on the filesystem. This option is useful when a filesystem of unknown origin is mounted.
nodev | Do not allow any special devices on the filesystem to be opened. This option is often used when a server exports device directories for a different architecture. The filesystem would be mounted with the nodev option on the server, since the values of the major and minor numbers are nonsensical to the server. The major and minor numbers are meaningful only on the clients that import them.
noatime | When reading a file, do not update its access time. This option is useful on filesystems where there are many files being frequently read and performance is more critical than updating the file access time (which is rarely ever important).
sync | Request that all I/O to the file system be done synchronously.

It is not necessary to unmount and remount the filesystem to change these flags; they may be changed while a filesystem is mounted. In addition, a filesystem that is mounted read-only can be upgraded to allow writing. Conversely, a filesystem that allows writing may be downgraded to read-only provided that no files are open for modification. The system administrator can forcibly downgrade the filesystem to read-only by requesting that any files open for writing have their access revoked.

Another service exported from the vnode interface is the ability to get information about a mounted filesystem. The staffs system call returns a buffer that gives the numbers of used and free disk blocks and inodes, along with the filesystem mount point, and the device, location, or program from which the filesystem is mounted. The getfsstat system call returns information about all the mounted filesystems. This interface avoids the need to track the set of mounted filesystems outside the kernel, as is done in many other UNIX variants.


6.6. Filesystem-Independent Services

The vnode interface not only supplies an object-oriented interface to the underlying filesystems but also provides a set of management routines that can be used by the client filesystems. These facilities are described in this section.

When the final file-entry reference to a file is closed, the usage count on the vnode drops to zero and the vnode interface calls the inactive() vnode operation. The inactive() call notifies the underlying filesystem that the file is no longer being used. The filesystem will often use this call to write dirty data back to the file but will not typically reclaim the memory holding file data. The filesystem is permitted to cache the file so that the latter can be reactivated quickly (i.e., without disk or network I/O) if the file is reopened.

In addition to the inactive() vnode operation being called when the reference count drops to zero, the vnode is placed on a systemwide free list. Unlike many vendor's vnode implementations, which have a fixed number of vnodes allocated to each filesystem type, the FreeBSD kernel keeps a single systemwide collection of vnodes. When an application opens a file that does not currently have an in-memory vnode, the client filesystem calls the getnewvnode() routine to allocate a new vnode. The kernel maintains two lists of free vnodes: those that have data pages cached in memory and those that do not have any data pages cached in memory. The preference is to reuse vnodes with no cached pages, since the reuse of a vnode with cached pages will cause all the cached pages associated with that vnode to lose their identity. If the vnodes were not classified separately, then an application that walked the filesystem tree doing stat calls on each file that it encountered would eventually flush all the vnodes referencing data pages, thus losing the identity of all the cached pages in the kernel. So when allocating a new vnode, the getnewvnode() routine first checks the front of the free list of vnodes with no cached pages and only if that list is empty does it select from the front of the list of vnodes with cached pages.

Having selected a vnode, the getnewvnode() routine then calls the vnode's reclaim() operation to notify the filesystem currently using that vnode that it is about to be reused. The reclaim() operation writes back any dirty data associated with the underlying object, removes the underlying object from any lists that it is on (such as hash lists used to find it), and frees up any auxiliary storage that was being used by the object. The vnode is then returned for use by the new client filesystem.

The benefit of having a single global vnode table is that the kernel memory dedicated to vnodes is used more efficiently than when several filesystem-specific collections of vnodes are used. Consider a system that is willing to dedicate memory for 1000 vnodes. If the system supports 10 filesystem types, then each filesystem type will get 100 vnodes. If most of the activity moves to a single filesystem (e.g., during the compilation of a kernel located in a local filesystem), all the active files will have to be kept in the 100 vnodes dedicated to that filesystem while the other 900 vnodes sit idle. In a FreeBSD system, all 1000 vnodes could be used for the active filesystem, allowing a much larger set of files to be cached in memory. If the center of activity moved to another filesystem (e.g., compiling a program on an NFS mounted filesystem), the vnodes would migrate from the previously active local filesystem over to the NFS filesystem. Here, too, there would be a much larger set of cached files than if only 100 vnodes were available using a partitioned set of vnodes.

The reclaim() operation is a disassociation of the underlying filesystem object from the vnode itself. This ability, combined with the ability to associate new objects with the vnode, provides functionality with usefulness that goes far beyond simply allowing vnodes to be moved from one filesystem to another. By replacing an existing object with an object from the dead filesystem-a filesystem in which all operations except close fail—the kernel revokes the object. Internally, this revocation of an object is provided by the vgone() routine.

This revocation service is used for session management, where all references to the controlling terminal are revoked when the session leader exits. Revocation works as follows. All open terminal descriptors within the session reference the vnode for the special device representing the session terminal. When vgone() is called on this vnode, the underlying special device is detached from the vnode and is replaced with the dead filesystem. Any further operations on the vnode will result in errors, because the open descriptors no longer reference the terminal. Eventually, all the processes will exit and will close their descriptors, causing the reference count to drop to zero. The inactive() routine for the dead filesystem returns the vnode to the front of the free list for immediate reuse because it will never be possible to get a reference to the vnode again.

The revocation service supports forcible unmounting of filesystems. If the kernel finds an active vnode when unmounting a filesystem, it simply calls the vgone() routine to disassociate the active vnode from the filesystem object. Processes with open files or current directories within the filesystem find that they have simply vanished, as though they had been removed. It is also possible to downgrade a mounted filesystem from read-write to read-only. Instead of access being revoked on every active file within the filesystem, only those files with a nonzero number of references for writing have their access revoked.

Finally, the ability to revoke objects is exported to processes through the revoke system call. This system call can be used to ensure controlled access to a device such as a pseudo-terminal port. First, the ownership of the device is changed to the desired user and the mode is set to owner-access only. Then the device name is revoked to eliminate any interlopers that already had it open. Thereafter, only the new owner is able to open the device.

The Name Cache

Name-cache management is another service that is provided by the vnode management routines. The interface provides a facility to add a name and its corresponding vnode, to lookup a name to get the corresponding vnode, and to delete a specific name from the cache. In addition to providing a facility for deleting specific names, the interface also provides an efficient way to invalidate all names that reference a specific vnode. Each vnode has a list that links together all their entries in the name cache. When the references to the vnode are to be deleted, each entry on the list is purged. Each directory vnode also has a second list of all the cache entries for names that are contained within it. When a directory vnode is to be purged, it must delete all the name-cache entries on this second list. A vnode's name-cache entries must be purged each time it is reused by getnewvnode() or when specifically requested by a client (e.g., when a directory is being renamed).

The cache-management routines also allow for negative caching. If a name is looked up in a directory and is not found, that name can be entered in the cache, along with a null pointer for its corresponding vnode. If the name is later looked up, it will be found in the name table, and thus the kernel can avoid scanning the entire directory to determine that the name is not there. If a name is added to a directory, then the name cache must lookup that name and purge it if it finds a negative entry. Negative caching provides a significant performance improvement because of path searching in command shells. When executing a command, many shells will look at each path in turn, looking for the executable. Commonly run executables will be searched for repeatedly in directories in which they do not exist. Negative caching speeds these searches.

Buffer Management

Historically, UNIX systems divided the main memory into two main pools. The first was the virtual-memory pool that was used to cache process pages. The second was the buffer pool and was used to cache filesystem data. The main memory was divided between the two pools when the system booted and there was no memory migration between the pools once they were created.

With the addition of the mmap system call, the kernel supported the mapping of files into the address space of a process. If a file is mapped in with the MAP_SHARED attribute, changes made to the mapped file are to be written back to the disk and should show up in read calls done by other processes. Providing these semantics is difficult if there are copies of a file in both the buffer cache and the virtual-memory cache. Thus, FreeBSD merged the buffer cache and the virtual-memory cache into a single-page cache.

As described in Chapter 5, virtual memory is divided into a pool of pages holding the contents of files and a pool of anonymous pages holding the parts of a process that are not backed by a file such as its stack and heap. Pages backed by a file are identified by their vnode and logical block number. Rather than rewrite all the filesystems to lookup pages in the virtual-memory pool, a buffer-cache emulation layer was written. The emulation layer has the same interface as the old buffer-cache routines but works by looking up the requested file pages in the virtual-memory cache. When a filesystem requests a block of a file, the emulation layer calls the virtual-memory system to see if it is in memory. If it is not in memory, the virtual-memory system arranges to have it read. Normally, the pages in the virtual-memory cache are not mapped into the kernel address space. However, a filesystem often needs to inspect the blocks that it requests—for example, if it is a directory or filesystem metadata. Thus, the buffer-cache emulation layer must not only find the requested block but also allocate some kernel address space and map the requested block into it. The filesystem then uses the buffer to read, write, or manipulate the data and when done releases the buffer. On release, the buffer may be held briefly but soon is dissolved by releasing the kernel mapping, dropping the reference count on the virtual-memory pages and releasing the header.

The virtual-memory system does not have any way to describe blocks that are identified as a block associated with a disk. A small remnant of the buffer cache remains to hold these disk blocks that are used to hold filesystem metadata such as superblocks, bitmaps, and inodes.

The internal kernel interface to the buffer-cache emulation layer is simple. The filesystem allocates and fills buffers by calling the bread() routine. Bread() takes a vnode, a logical block number, and a size, and returns a pointer to a locked buffer. The details on how a buffer is created are given in the next subsection. Any other thread that tries to obtain the buffer will be put to sleep until the buffer is released.

A buffer can be released in one of four ways. If the buffer has not been modified, it can simply be released through use of brelse(), which checks for any threads that are waiting for it. If any threads are waiting, they are awakened. Otherwise the buffer is dissolved by returning its contents back to the virtual-memory system, releasing its kernel address-space mapping and releasing the buffer.

If the buffer has been modified, it is called dirty. Dirty buffers must eventually be written back to their filesystem. Three routines are available based on the urgency with which the data must be written. In the typical case, bdwrite() is used. Since the buffer may be modified again soon, it should be marked as dirty but should not be written immediately. After the buffer is marked as dirty, it is returned to the dirty-buffer list and any threads waiting for it are awakened. The heuristic is that, if the buffer will be modified again soon, the I/O would be wasted. Because the buffer is typically held for 20 to 30 seconds before it is written, a thread doing many small writes will not repeatedly access the disk or network.

If a buffer has been filled completely, then it is unlikely to be written again soon, so it should be released with bawrite(). Bawrite() schedules an I/O on the buffer but allows the caller to continue running while the output completes.

The final case is bwrite(), which ensures that the write is complete before proceeding. Because bwrite() can introduce a long latency to the writer, it is used only when a process explicitly requests the behavior (such as the fsync system call), when the operation is critical to ensure the consistency of the filesystem after a system crash, or when a stateless remote filesystem protocol such as NFS is being served. A buffer that is written using bawrite() or bwrite() is placed on the appropriate output queue. When the output completes, the brelse() routine is called to awaken any threads that are waiting for it or, if there is no immediate need for it, to dissolve the buffer.

Some buffers, though clean, may be needed again soon. To avoid the overhead of repeatedly creating and dissolving buffers, the buffer-cache emulation layer provides the bqrelse() routine to let the filesystem notify it that it expects to use the buffer again soon. The bqrelse() routine places the buffer on a clean list rather than dissolving it.

Figure 6.8 shows a snapshot of the buffer pool. A buffer with valid contents is contained on exactly one bufhash hash chain. The kernel uses the hash chains to determine quickly whether a block is in the buffer pool, and if it is, to locate it. A buffer is removed only when its contents become invalid or it is reused for different data. Thus, even if the buffer is in use by one thread, it can still be found by another thread, although it will be locked so that it will not be used until its contents are consistent.

Figure 6.8. Snapshot of the buffer pool. Key: V—vnode; X—file offset.

In addition to appearing on the hash list, each unlocked buffer appears on exactly one free list. The first free list is the LOCKED list. Buffers on this list cannot be flushed from the cache. This list was originally intended to hold superblock data; in FreeBSD, it holds only buffers being written in background. In a background write, the contents of a dirty buffer are copied to another anonymous buffer. The anonymous buffer is then written to disk. The original buffer can continue to be used while the anonymous buffer is being written. Background writes are used primarily for fast and continuously changing blocks such as those that hold filesystem allocation bitmaps. If the block holding the bitmap was written normally, it would be locked and unavailable while it waited on the disk queue to be written. Thus, applications trying to write files in the area described by the bitmap would be blocked from running while they waited for the write of the bitmap to finish so that they could update the bitmap. By using background writes for bitmaps, applications are rarely forced to wait to update a bitmap.

The second list is the DIRTY list. Buffers that have been modified, but not yet written to disk, are stored on this list. The DIRTY list is managed using a least-recently used algorithm. When a buffer is found on the DIRTY list, it is removed and used. The buffer is then returned to the end of the DIRTY list. When too many buffers are dirty, the kernel starts the buffer daemon running. The buffer daemon writes buffers starting from the front of the DIRTY list. Thus, buffers written repeatedly will continue to migrate to the end of the DIRTY list and are not likely to be prematurely written or reused for new blocks.

The third free list is the CLEAN list. This list holds blocks that a filesystem is not currently using but that it expects to use soon. The CLEAN list is also managed using a least-recently used algorithm. If a requested block is found on the CLEAN list, it is returned to the end of the list.

The final list is the list of empty buffers—the EMPTY list. The empty buffers are just headers and have no memory associated with them. They are held on this list waiting for another mapping request.

When a new buffer is needed, the kernel first checks to see how much memory is dedicated to the existing buffers. If the memory in use is below its permissible threshold, a new buffer is created from the EMPTY list. Otherwise the oldest buffer is removed from the front of the CLEAN list. If the CLEAN list is empty, the buffer daemon is awakened to clean up and release a buffer from the DIRTY list.

Implementation of Buffer Management

Having looked at the functions and algorithms used to manage the buffer pool, we now turn our attention to the implementation requirements for ensuring the consistency of the data in the buffer pool. Figure 6.9 shows the support routines that implement the interface for getting buffers. The primary interface to getting a buffer is through bread(), which is called with a request for a data block of a specified size for a specified vnode. There is also a related interface, breadn(), that both gets a requested block and starts read-ahead for additional blocks. Bread() first calls getblk() to find out whether the data block is available in an existing buffer. If the block is available in a buffer, getblk() calls bremfree() to take the buffer off whichever free list it is on and to lock it; bread() can then return the buffer to the caller.

Figure 6.9. Procedural interface to the buffer-allocation system.

[View full size image]

If the block is not already in an existing buffer, getblk() calls getnewbuf() to allocate a new buffer, using the algorithm described in the previous subsection. The new buffer is then passed to allocbuf(), which is responsible for determining how to constitute the contents of the buffer.

The common case is that the buffer is to contain a logical block of a file. Here, allocbuf() must request the needed block from the virtual-memory system. If the virtual-memory system does not already have the needed block, it arranges to get it brought into its page cache. The allocbuf() routine then allocates an appropriately sized piece of kernel address space and requests the virtual-memory system to map the needed file block into that address space. The buffer is then marked filled and returned through getblk() and bread().

The other case is that the buffer is to contain a block of filesystem metadata such as a bitmap or an inode block that is associated with a disk device rather than a file. Because the virtual memory does not (currently) have any way to track such blocks, they can be held in memory only within buffers. Here, allocbuf() must call the kernel malloc() routine to allocate memory to hold the block. The allocbuf() routine then returns the buffer to getblk() and bread() marked busy and unfilled. Noticing that the buffer is unfilled, bread() passes the buffer to the strategy() routine for the underlying filesystem to have the data read in. When the read completes, the buffer is returned.

To maintain the consistency of the filesystem, the kernel must ensure that a disk block is mapped into at most one buffer. If the same disk block were present in two buffers, and both buffers were marked dirty, the system would be unable to determine which buffer had the most current information. Figure 6.10 shows a sample allocation. In the middle of the figure are the blocks on the disk. Above the disk is shown an old buffer containing a 4096-byte fragment for a file that presumably has been removed or shortened. The new buffer is going to be used to hold a 4096-byte fragment for a file that is presumably being created and that will reuse part of the space previously held by the old file. The kernel maintains the consistency by purging old buffers when files are shortened or removed. Whenever a file is removed, the kernel traverses its list of dirty buffers. For each buffer, the kernel cancels its write request and dissolves the buffer so that the buffer cannot be found in the buffer pool again. For a file being partially truncated, only the buffers following the truncation point are invalidated. The system can then allocate the new buffer, knowing that the buffer maps the corresponding disk blocks uniquely.

Figure 6.10. Potentially overlapping allocation of buffers.


6.7. Stackable Filesystems

The early vnode interface was simply an object-oriented interface to an underlying filesystem. As the demand grew for new filesystem features, it became desirable to find ways of providing them without having to modify the existing and stable filesystem code. One approach is to provide a mechanism for stacking several filesystems on top of one another other [Rosenthal, 1990]. The stacking ideas were refined and implemented in the 4.4BSD system [Heidemann & Popek, 1994]. The implementation of the stacking has been refined in FreeBSD, but the semantics remain largely unchanged from those found in 4.4BSD. The bottom of a vnode stack tends to be a disk-based filesystem, whereas the layers used above it typically transform their arguments and pass on those arguments to a lower layer.

In all UNIX systems, the mount command takes a special device as a source and maps that device onto a directory mount point in an existing filesystem. When a filesystem is mounted on a directory, the previous contents of the directory are hidden; only the contents of the root of the newly mounted filesystem are visible. To most users, the effect of the series of mount commands done at system startup is the creation of a single seamless filesystem tree.

Stacking also uses the mount command to create new layers. The mount command pushes a new layer onto a vnode stack; an unmount command removes a layer. Like the mounting of a filesystem, a vnode stack is visible to all processes running on the system. The mount command identifies the underlying layer in the stack, creates the new layer, and attaches that layer into the filesystem name space. The new layer can be attached to the same place as the old layer (covering the old layer) or to a different place in the tree (allowing both layers to be visible). An example is shown in the next subsection.

If layers are attached to different places in the name space, then the same file will be visible in multiple places. Access to the file under the name of the new layer's name space will go to the new layer, whereas that under the old layer's name space will go to only the old layer.

When a file access (e.g., an open, read, stat, or close) occurs to a vnode in the stack, that vnode has several options:

  • Do the requested operations and return a result.

  • Pass the operation without change to the next-lower vnode on the stack. When the operation returns from the lower vnode, it may modify the results or simply return them.

  • Modify the operands provided with the request and then pass it to the next-lower vnode. When the operation returns from the lower vnode, it may modify the results, or simply return them.

If an operation is passed to the bottom of the stack without any layer taking action on it, then the interface will return the error "operation not supported."

Vnode interfaces released before 4.4BSD implemented vnode operations as indirect function calls. The requirements that intermediate stack layers bypass operations to lower layers and that new operations can be added into the system at boot or module load time mean that this approach is no longer adequate. Filesystems must be able to bypass operations that may not have been defined at the time that the filesystem was implemented. In addition to passing through the function, the filesystem layer must also pass through the function parameters, which are of unknown type and number.

To resolve these two problems in a clean and portable way, the kernel places the vnode operation name and its arguments into an argument structure. This argument structure is then passed as a single parameter to the vnode operation. Thus, all calls on a vnode operation will always have exactly one parameter, which is the pointer to the argument structure. If the vnode operation is one that is supported by the filesystem, then it will know what the arguments are and how to interpret them. If it is an unknown vnode operation, then the generic bypass routine can call the same operation in the next-lower layer, passing to the operation the same argument structure that it received. In addition, the first argument of every operation is a pointer to the vnode operation description. This description provides to a bypass routine the information about the operation, including the operation's name and the location of the operation's parameters. An example access-check call and its implementation for the UPS filesystem are shown in Figure 6.11. Note that the vop_access_args structure is normally declared in a header file, but here it is declared at the function site to simplify the example.

Figure 6.11. Call to and function header for access vnode operation.
{
    . . .
    /*
     * Check for read permission on file ''vp''.
     */
    if (error = VOP_ACCESS(vp, VREAD, cred, td))
        return (error);
    . . .
}

/*
 * Check access permission for a file.
 */
int ufs_access(
    struct vop_access_args {
        struct vnodeop_desc *a_desc; /* operation descrip. */
        struct vnode *a_vp;          /* file to be checked */
        int a_mode;                  /* access mode sought */
        struct ucred *a_cred;        /* user seeking access */
        struct thread *a_td;         /* associated thread */
    } *ap);
{

    if (permission granted)
        return (1);
    return (0);
}

Simple Filesystem Layers

The simplest filesystem layer is nullfs. It makes no transformations on its arguments, simply passing through all requests that it receives and returning all results that it gets back. Although it provides no useful functionality if it is simply stacked on top of an existing vnode, nullfs can provide a loopback filesystem by mounting the filesystem rooted at its source vnode at some other location in the filesystem tree. The code for nullfs is also an excellent starting point for designers who want to build their own filesystem layers. Examples that could be built include a compression layer or an encryption layer.

A sample vnode stack is shown in Figure 6.12. The figure shows a local filesystem on the bottom of the stack that is being exported from /local via an NFS layer. Clients within the administrative domain of the server can import the /local filesystem directly because they are all presumed to use a common mapping of UIDs to user names.

Figure 6.12. Stackable vnodes.

The umapfs filesystem works much like the nullfs filesystem in that it provides a view of the file tree rooted at the /local filesystem on the /export mount point. In addition to providing a copy of the /local filesystem at the /export mount point, it transforms the credentials of each system call made to files within the /export filesystem. The kernel does the transformation using a mapping that was provided as part of the mount system call that created the umapfs layer.

The /export filesystem can be exported to clients from an outside administrative domain that uses different UIDs and GIDs. When an NFS request comes in for the /export filesystem, the umapfs layer modifies the credential from the foreign client by mapping the UIDs used on the foreign client to the corresponding UIDs used on the local system. The requested operation with the modified credential is passed down to the lower layer corresponding to the /local filesystem, where it is processed identically to a local request. When the result is returned to the mapping layer, any returned credentials are mapped inversely so that they are converted from the local UIDs to the outside UIDs, and this result is sent back as the NFS response.

There are three benefits to this approach:

  1. There is no cost of mapping imposed on the local clients.

  2. There are no changes required to the local filesystem code or the NFS code to support mapping.

  3. Each outside domain can have its own mapping. Domains with simple mappings consume small amounts of memory and run quickly; domains with large and complex mappings can be supported without detracting from the performance of simpler environments.

Vnode stacking is an effective approach for adding extensions, such as the umapfs service.

The Union Filesystem

The union filesystem is another example of a middle filesystem layer. Like the nullfs, it does not store data but just provides a name-space transformation. It is loosely modeled on the work on the 3-D filesystem [Korn & Krell, 1989], on the Translucent filesystem [Hendricks, 1990], and on the Automounter [Pendry & Williams, 1994]. The union filesystem takes an existing filesystem and transparently overlays the latter on another filesystem. Unlike most other filesystems, a union mount does not cover up the directory on which the filesystem is mounted. Instead, it shows the logical merger of both directories and allows both directory trees to be accessible simultaneously [Pendry & McKusick, 1995].

A small example of a union-mount stack is shown in Figure 6.13. Here, the bottom layer of the stack is the src filesystem that includes the source for the shell program. Being a simple program, it contains only one source and one header file. The upper layer that has been union mounted on top of src initially contains just the src directory. When the user changes directory into shell, a directory of the same name is created in the top layer. Directories in the top layer corresponding to directories in the lower layer are created only as they are encountered while the top layer is traversed. If the user were to run a recursive traversal of the tree rooted at the top of the union-mount location, the result would be a complete tree of directories matching the underlying filesystem. In our example, the user now types make in the shell directory. The sh executable is created in the upper layer of the union stack. To the user, a directory listing shows the sources and executable all apparently together, as shown on the right in Figure 6.13.

Figure 6.13. A union-mounted filesystem. The /usr/src filesystem is on the bottom, and the /tmp/src filesystem is on the top.

All filesystem layers, except the top one, are treated as though they were readonly. If a file residing in a lower layer is opened for reading, a descriptor is returned for that file. If a file residing in a lower layer is opened for writing, the kernel first copies the entire file to the top layer and then returns a descriptor referencing the copy of the file. The result is that there are two copies of the file: the original unmodified file in the lower layer and the modified copy of the file in the upper layer. When the user does a directory listing, any duplicate names in the lower layer are suppressed. When a file is opened, a descriptor for the file in the uppermost layer in which the name appears is returned. Thus, once a file has been copied to the top layer, instances of the file in lower layers become inaccessible.

The tricky part of the union filesystem is handling the removal of files that reside in a lower layer. Since the lower layers cannot be modified, the only way to remove a file is to hide it by creating a whiteout directory entry in the top layer. A whiteout is an entry in a directory that has no corresponding file; it is distinguished by having an inode number of 1. If the kernel finds a whiteout entry while searching for a name, the lookup is stopped and the "no such file or directory" error is returned. Thus, the file with the same name in a lower layer appears to have been removed. If a file is removed from the top layer, it is necessary to create a whiteout entry for it only if there is a file with the same name in the lower level that would reappear.

When a process creates a file with the same name as a whiteout entry, the whiteout entry is replaced with a regular name that references the new file. Because the new file is being created in the top layer, it will mask out any files with the same name in a lower layer. When a user does a directory listing, white-out entries and the files that they mask usually are not shown. However, there is an option that causes them to appear.

One feature that has long been missing in UNIX systems is the ability to recover files after they have been deleted. For the union filesystem, the kernel can implement file recovery trivially simply by removing the whiteout entry to expose the underlying file. For filesystems that provide file recovery, users can recover files by using a special option to the remove command. Processes can recover files by using the undelete system call.

When a directory whose name appears in a lower layer is removed, a whiteout entry is created just as it would be for a file. However, if the user later attempts to create a directory with the same name as the previously deleted directory, the union filesystem must treat the new directory specially to avoid having the previous contents from the lower-layer directory reappear. When a directory that replaces a whiteout entry is created, the union filesystem sets a flag in the directory metadata to show that this directory should be treated specially. When a directory scan is done, the kernel returns information about only the top-level directory; it suppresses the list of files from the directories of the same name in the lower layers.

The union filesystem can be used for many purposes:

  • It allows several different architectures to build from a common source base. The source pool is NFS mounted onto each of several machines. On each host machine, a local filesystem is union mounted on top of the imported source tree. As the build proceeds, the objects and binaries appear in the local filesystem that is layered above the source tree. This approach not only avoids contaminating the source pool with binaries, but also speeds the compilation, because most of the filesystem traffic is on the local filesystem.

  • It allows compilation of sources on read-only media such as CD-ROMs. A local filesystem is union mounted above the CD-ROM sources. It is then possible to change into directories on the CD-ROM and to give the appearance of being able to edit and compile in that directory.

  • It allows creation of a private source directory. The user creates a source directory in her own work area and then union mounts the system sources underneath that directory. This feature is possible because the restrictions on the mount command have been relaxed. If the sysctl vfs.usermount option has been enabled, any user can do a mount if she owns the directory on which the mount is being done and she has appropriate access permissions on the device or directory being mounted (read permission is required for a read-only mount, read-write permission is required for a read-write mount). Only the user who did the mount or the superuser can unmount a filesystem.

Other Filesystems

There are several other filesystems included as part of FreeBSD. The portal filesystem mounts a process onto a directory in the file tree. When a pathname that traverses the location of the portal is used, the remainder of the path is passed to the process mounted at that point. The process interprets the path in whatever way it sees fit, then returns a descriptor to the calling process. This descriptor may be for a socket connected to the portal process. If it is, further operations on the descriptor will be passed to the portal process for the latter to interpret. Alternatively, the descriptor may be for a file elsewhere in the filesystem.

Consider a portal process mounted on /dialout used to manage a bank of dialout modems. When a process wanted to connect to an outside number, it would open /dialout/15105551212/28800 to specify that it wanted to dial 1-510-555-1212 at 28800 baud. The portal process would get the final two pathname components. Using the final component, it would determine that it should find an unused 28800-baud modem. It would use the other component as the number to which to place the call. It would then write an accounting record for future billing, and would return the descriptor for the modem to the process.

An interesting use of the portal filesystem is to provide an Internet service directory. For example, with an Internet portal process mounted on /net, an open of /net/tcp/McKusick.COM/smtp returns a TCP socket descriptor to the calling process that is connected to the SMTP server on McKusick.COM. Because access is provided through the normal filesystem, the calling process does not need to be aware of the special functions necessary to create a TCP socket and to establish a TCP connection [Stevens & Pendry, 1995].

There are several filesystems that are designed to provide a convenient interface to kernel information. The procfs filesystem is normally mounted at /proc and provides a view of the running processes in the system. Its primary use is for debugging, but it also provides a convenient interface for collecting information about the processes in the system. A directory listing of /proc produces a numeric list of all the processes in the system. The /proc interface is more fully described in Section 4.9.

The fdesc filesystem is normally mounted on /dev/fd and provides a list of all the active file descriptors for the currently running process. An example where this is useful is specifying to an application that it should read input from its standard input. Here, you can use the pathname /dev/fd/0 instead of having to come up with a special convention, such as using the name - to tell the application to read from its standard input.

The kernfs filesystem is normally mounted on /kern and contains files that have various information about the system. It includes information such as the host name, time of day, and version of the system.

Finally there is the cd9660 filesystem. It allows ISO-9660-compliant filesystems, with or without Rock Ridge extensions, to be mounted. The ISO-9660 filesystem format is most commonly used on CD-ROMs.


Exercises

6.1 Where are the read and write attributes of an open file descriptor stored?

6.2 Why is the close-on-exec bit located in the per-process descriptor table instead of in the system file table?

6.3 Why are the file-table entries reference counted?

6.4 What three shortcomings of lock files are addressed by the FreeBSD descriptor-locking facilities?

6.5 What two problems are raised by mandatory locks?

6.6 Why is the implementation of select split between the descriptor-management code and the lower-level routines?

6.7 Describe how the process selecting flag is used in the implementation of select.

6.8 The syncer daemon starts as part of system boot. Once every second, it does an fsync on any vnodes that it finds that have been dirty for 30 seconds. What problem could arise if this daemon were not run?

6.9 When is a vnode placed on the free list?

6.10 Why must the lookup routine call through the vnode interface once for each component in a pathname?

6.11 Give three reasons for revoking access to a vnode.

6.12 Why are the buffer headers allocated separately from the memory that holds the contents of the buffer?

6.13 Asynchronous I/O is provided through the aio_read and aio_write systems calls rather than through the traditional read and write system calls. What problems arise with providing asynchronous I/O in the existing read-write interface?

*6.14 Why are there both a CLEAN list and a DIRTY list instead of all buffers being managed on one list?

*6.15 If a process reads a large file, the blocks of the file will fill the virtual memory cache completely, flushing out all other contents. All other processes in the system then will have to go to disk for all their filesystem accesses. Write an algorithm to control the purging of the buffer cache.

*6.16 Vnode operation parameters are passed between layers in structures. What alternatives are there to this approach? Explain why your approach is more or less efficient, compared to the current approach, when there are less than five layers in the stack. Also compare the efficiency of your solution when there are more than five layers in the stack.


References

Accetta et al., 1986. M. Accetta, R. Baron, W. Bolosky, D. Golub, R. Rashid, A. Tevanian, & M. Young, "Mach: A New Kernel Foundation for UNIX Development," USENIX Association Conference Proceedings, pp. 93-113, June 1986.

Heidemann & Popek, 1994. J. S. Heidemann & G. J. Popek, "File-System Development with Stackable Layers," ACM Transactions on Computer Systems, vol. 12, no. 1, pp. 58-89, February 1994.

Hendricks, 1990. D. Hendricks, "A Filesystem for Software Development," USENIX Association Conference Proceedings, pp. 333-340, June 1990.

Korn & Krell, 1989. D. Korn & E. Krell, "The 3-D File System," USENIX Association Conference Proceedings, pp. 147-156, June 1989.

Lemon, 2001. J. Lemon, "Kqueue: A Generic and Scalable Event Notification Facility," Proceedings of the Freenix Track at the 2001 Usenix Annual Technical Conference, pp. 141-154, June 2001.

Pendry & McKusick, 1995. J. Pendry & M. K. McKusick, "Union Mounts in 4.4BSD-Lite," USENIX Association Conference Proceedings, pp. 25-33, January 1995.

Pendry & Williams, 1994. J. Pendry & N. Williams, "AMD: The 4.4BSD Automounter Reference Manual," in 4.4BSD System Manager's Manual, pp. 13:1-57, O'Reilly & Associates, Inc., Sebastopol, CA, 1994.

Peterson, 1983. G. Peterson, "Concurrent Reading While Writing," ACM Transactions on Programming Languages and Systems, vol. 5, no. 1, pp. 46-55, January 1983.

Rosenthal, 1990. D. Rosenthal, "Evolving the Vnode Interface," USENIX Association Conference Proceedings, pp. 107-118, June 1990.

Stevens & Pendry, 1995. R. Stevens & J. Pendry, "Portals in 4.4BSD," USENIX Association Conference Proceedings, pp. 1-10, January 1995.


Chapter 7. Devices


7.1. Device Overview

This chapter describes the part of the system that interfaces to the hardware as is shown in the bottom part of Figure 6.1 (on page 216). Historically, the device interface was static and simple. Devices were discovered as the system was booted and did not change thereafter. Filesystems were built in a partition of a single disk. When a disk driver received a request from a filesystem to write a block, it would add the base offset of the partition and do a bounds check based on information from its disk label. It would then do the requested I/O and return the result or error to the filesystem. A typical disk driver could be written in a few hundred lines of code.

As the system has evolved, the complexity of the I/O system has increased with the addition of new functionality. The new functionality can be broken into two categories:

  • Disk management

  • I/O routing and control

Each of these areas is handled by a new subsystem in FreeBSD.

Disk management consists of organizing the myriad ways that disks can be used to build a filesystem. A disk may be broken up into several slices, each of which can be used to support a different operating system. Each of these slices may be further subdivided into partitions that can be used to support filesystems as they did historically. However, it is also possible to combine several slices and/or partitions to create a virtual partition on which to build a filesystem that spans several disks. The virtual partition may concatenate together several partitions to stripe the filesystem across several disks, thus providing a high-bandwidth filesystem. Or the underlying partitions may be put together in a RAID (Redundant Array of Inexpensive Disks) to provide a higher level of reliability and accessibility than a disk alone. Or the partitions may be organized into two equal-sized groups and mirrored to provide an even higher level of reliability and accessibility than RAID. The aggregation of physical disk partitions into a virtual partition in these ways is referred to as volume management.

Rather than building all this functionality into all the filesystems or disk drivers, it has been abstracted out into the GEOM (geometry) layer. The GEOM layer takes as input the set of disks available on the system. It is responsible for doing volume management. At a low level volume management creates, maintains, and interprets the slice tables and the disk labels defining the partitions within each slice. At a higher level, GEOM combines the physical disk partitions through striping, RAID, or mirroring to create the virtual partitions that are exported to the filesystem layer above. The virtual partition appears to the filesystem as a single large disk. As the filesystem does I/O within the virtual partition, the GEOM layer determines which disk(s) are involved and breaks up and dispatches the I/O request to the appropriate physical drives. The operation of the GEOM layer is described in Section 7.2.

The PC I/O Architecture

Historically, architectures had only one or two I/O busses and types of disk controllers. As described in the next subsection, a modern PC today can have several types of disks connected to the machine through five or more different types of interfaces. The complexity of the disk controllers themselves rivals that of the entire early UNIX operating system. Early controllers could only handle one disk I/O at a time. Today's controllers can typically juggle up to 64 simultaneous requests through a scheme called tagged queueing. A request works its way through the controller being posted as it is received, scheduled to be done, completed, and reported back to the requester. I/O may also be cached in the controller to allow future requests to be handled more quickly. Another task handled by the controller is to provide a replacement for a disk sector with a permanent error with an alternate good sector.

The current PC I/O architecture is shown in Figure 7.1. Far greater detail is available at Arch [2003]. On the left of the figure is the CPU that has a high-speed interconnect through the northbridge to the system's main memory and the graphics memory that drives the system display. Note that the L1 and L2 caches are not shown in this picture because they are considered as part of the CPU. Continuing to the right of the northbridge is a moderate-speed connection to the southbridge. The southbridge connects all the high-throughput I/O busses to the system. These busses include the following:

  • The PCI (Peripheral Component Interconnect) bus. The PCI-X and PCI Express busses are modern, backward-compatible implementations of the PCI standard. Most modern I/O cards are designed to plug into some variant of the PCI bus, since this bus architecture provides high-throughput speeds and is well designed for fully automated autoconfiguration. This bus also has the advantage of being available on many other computer architectures besides the PC.

  • The ATA (Advanced Technology Attachment) bus. The original ATA parallel bus remains as a vestige of earlier PC designs. It supports a maximum of two drives per channel, does not have hot-plug support (changing of drives while the system is powered up and operational), and has a maximum transfer speed of 133 Mbyte per second compared with the maximum transfer speed of 320 Mbyte per second of the SCSI (Small Computer System Interface) parallel interface. ATA parallel support remains because of the many cheap disk drives that are available to connect to it. The newer Serial ATA adds hot-plug support and transfers data at rates comparable to the SCSI parallel interface.

  • The USB (Universal Serial Bus) bus. The USB bus provides a moderate speed (up to 60 Mbyte per second) input typically used for video cameras, memory cards, scanners, printers, and some of the newer human input devices such as keyboards, mice, and joysticks.

  • The Firewire (IEEE 1394) bus. Firewire is faster than the USB bus, running at up to 100 Mbyte per second. It is used by memory-card readers, external disks, and some professional digital cameras. Firewire was invented by Apple Computer but is now commonly found on PC-architecture machines.

  • CardBus cards, or its slower cousin, PC cards, historically known as PCMCIA cards (Personal Computer Memory Card International Association). CardBus cards are the ubiquitous devices about the size of a thick credit card that are primarily used in laptop computers to provide Ethernet, modem, FAX, 802.11 wireless access, minidisk, SCSI and ATA disk controllers, and many other functions.

  • The PIC (Programmable Interrupt Controller). The PIC maps the device interrupts to IRQ (Interrupt ReQuest) values for the CPU. Most modern machines use an IOAPIC (I/O Advanced Programmable Interrupt Controller) that provides much finer-grain control over the device interrupts. All processors since the Pentium Pro (1997) have had an LAPIC (Local Advanced Programmable Interrupt Controller) that works with the IOAPIC to support distribution of interrupts among the CPUs.

Figure 7.1. The PC I/O architecture. Key: ATA—Advanced Technology Attachment; USB—Universal Serial Bus; PCI—Peripheral Component Interconnect; (A)PIC— (Advanced) Programmable Interrupt Controller.

Also hanging off the southbridge is the Super I/O chip that provides slow-speed access to many legacy PC interfaces. These interfaces include the following:

  • The PS2 keyboard and mouse ports. Newer machines are connecting the keyboard and mouse through the USB port, but many legacy PS2 keyboards and mice remain.

  • Support for the AC97 (Audio CODEC) sound standard. This standard allows a single DSP (Digital Signal Processor) to be used to support both a modem and sound.

  • The embedded controller. The embedded controller is present on many mobile systems and controls various components, including power and sleep buttons, back-light intensity of the screen, and status lights. On modern systems, the embedded controller is accessed via the ACPI (Advanced Configuration and Power Interface). The ACPI standard evolves the existing collection of power management BIOS (Basic Input Output System) code, numerous APM (Advanced Power Management) APIs, (Application Programming Interface) and PNPBIOS (Plug and Play BIOS) APIs into a well-specified power management and configuration mechanism. The ACPI specification provides support for an orderly transition from legacy hardware to ACPI hardware and allows for multiple mechanisms to exist in a single machine and be used as needed [ACPI, 2003].

  • Support for legacy ISA (Industry Standard Architecture) bus-based peripherals. The ISA bus was the mainstay of the PC for many years and many embedded systems still use this bus. Because of their small size and formerly high volume, these peripherals tend to be cheap. Today they are relegated to slow functions such as dialup modem, 10-Mbit Ethernet, and 16-bit sound cards.

The Structure of the FreeBSD Mass Storage I/O Subsystem

There were several disk subsystems in early versions of FreeBSD. The first support for ATA and SCSI disks came from Mach 2.5. Both of these were highly device-specific. Efforts to replace both resulted in CAM (Common Access Method), introduced in FreeBSD 3.0, and ATA, introduced in FreeBSD 4.0. As the ATA effort was proceeding, the CAM maintainers attempted to have it become a CAM attachment. However, the strange reservation and locking rules of the ATA register file model was a poor match for the CAM implementation, so the ATA implementation with one exception remained separate.

CAM is an ANSI (American National Standards Institute) standard (X3.232-1996). A revised and enhanced version of CAM was proposed by the X3T10 group but was never approved [ANSI, 2002]. Although primarily used for SCSI, CAM is a way of interfacing IIBA (Host Bus Adapter) drivers (in CAM terminology SIM (Software Interface Module) drivers), midlayer transport glue, and peripheral drivers. CAM seems unlikely to ever be approved as a standard, but it still provides a useful framework for implementing a storage subsystem.

The FreeBSD CAM implementation supports SPI (SCSI Parallel Interface), Fibre Channel [ANSI, 2003], UMASS (USB Mass Storage), IEEE 1394 (Firewire), and ATAPI (Advanced Technology Attachment Packet Interface). It has peripheral drivers for disks (da), cdrom (cd), tapes (sa), tape changers (ch), processor devices (pt), and enclosure services (scs). Additionally, there is the target emulator that allows a computer to emulate any of the supported devices and a pass-through interface that allows user applications to send I/O requests to any CAM-controlled peripheral. The operation of the CAM layer is described in Section 7.3.

ATA devices have their own parallel system that is more integrated but less flexible than CAM. It implements its own midlayer transport that supports ATA disks (ad), RAID arrays (ar), cdrom (acd), floppy disk (afd), and tape (ast). It also supports operation of ATA hardware drivers connected to CardBus or the PCI bus, and the ISA bus. At the time of this writing, ATA has implemented fine-grained locking for multiprocessor support, which is not yet complete for CAM. The operation of the ATA layer is described in Section 7.4.

The structure of the FreeBSD Disk I/O subsystem is shown in Figure 7.2. As the figure shows, disk drives may be attached to the system through many busses.

Figure 7.2. The structure of the FreeBSD disk I/O subsystem.

[View full size image]

The fastest and most expensive connection is through a fiberoptic link. Such disk systems are usually used on large servers or when the data must travel farther than just within the case of the computer or to an adjacent rack.

The more common fast choice is a controller that plugs into the PCI bus such as an Adaptec parallel SCSI controller, which can support up to 15 disks and tapes. SCSI disks generally are faster and more reliable under heavy load than the more consumer-desktop-oriented ATA disks.

Parallel interface ATA disks are the cheapest and most ubiquitous. The serial interface ATA disks have faster access and have capabilities that rival those of their more costly SCSI cousins. The beginning of CAM support for ATA has begun to show up as CAM can operate ATAPI disk and tape drives. The ATAPI protocol is a subset of the SCSI MMC (Multi Media Command) specification and is used only for CD-ROM, DVD, and tape drives.

Disks may also be connected through the other busses available on the PC architecture. These include the CardBus, Firewire, and USB. Usually the disks are connected through an interface that acts as a bridge from the interface to a PCI or ISA bus. The CardBus, Firewire, and USB busses may also support other types of devices that will be directly connected to their device drivers rather than be managed by the CAM layer.

Network device drivers provide another important piece of functionality within the kernel. Rather than discuss them in this chapter, we shall defer describing them until we begin discussing the network stack in Section 12.1.

Autoconfiguration is the procedure carried out by the system to recognize and enable the hardware devices present in a system. Historically, autoconfiguration was done just once when the system was booted. In current machines, particularly portable machines such as laptop computers, devices routinely come and go while the machine is operating. Thus, the kernel must be prepared to configure, initialize, and make available hardware when it arrives and to drop operations with hardware that has departed. FreeBSD uses a device-driver infrastructure called newbus to manage the devices on the system. Newbus builds a tree rooted at an abstract root0 node and descending in a treelike structure down the various I/O paths and terminating at the various devices connected to the machine. On a uniprocessor system the root0 node is synonymous with the CPU. On a multiprocessor system the root0 node is logically connected to each of the CPUs. At the time of this writing not all devices have been brought in under newbus. Not yet supported, but soon to be added are CPUs, timers, and PICs. Most drivers that do not understand newbus are for old legacy devices for which support is likely to be dropped. The most notable example of missing support is for disks that have problems because they may be multihomed (e.g., accessible through multiple busses). Newbus is currently designed to operate as a tree structure. It will need to be extended to allow a directed acyclic graph structure before it will be able to support multi-homed disks. Device autoconfiguration is described in Section 7.5, which gives the details of configuring devices when they appear and cleaning up after them when they disappear.

Device Naming and Access

Historically, FreeBSD has used static device nodes located in /dev to provide access to the hardware devices on the system. This approach has several problems:

  • The device nodes are persistent entities in the filesystem and do not necessarily represent the hardware that is really connected to and available on the machine.

  • When new hardware is added to the kernel, the system administrator needs to create new device nodes to access the hardware.

  • If the hardware is later removed, the device nodes remain even though they are no longer usable.

  • Device nodes require coordination of the major and minor numbering scheme between the device-driver tables in the kernel and the shell scripts that create them.

FreeBSD 5.2 replaces the static /dev directory with a new DEVFS filesystem that is mounted on /dev when the kernel is booted. As devices are discovered either at boot or while the system is running, their names appear in the /dev filesystem. When a device disappears or becomes unavailable, its entries in /dev disappear. DEVFS has several benefits over the old static /dev directory:

  • Only devices that are currently available appear in /dev.

  • Adding a device to the system causes its device nodes to appear in /dev, obviating the need for a system administrator to create new device nodes.

  • It is no longer necessary to coordinate device major and minor numbers between the kernel and device-creation scripts or filesystem device nodes.

One benefit of the old static /dev was that device nodes could be given non-standard names, access permissions, owners, or groups. To provide the same flexibility, DEVFS has a rule-set mechanism that allows these changes to be automated in the new /dev implementation. These rule sets can be put in place when the system is booted and can be created or modified at any time that the system is running. Each rule provides a pattern to identify the device nodes to be affected. For each matched device node it specifies one or more actions that should be taken. Actions include creating a symbolic link to provide a nonstandard name as well as setting nonstandard permissions, owner, or group. The rule sets are checked and applied whenever a new device node is created or destroyed. They may also be checked and applied when explicitly requested to do so by the system administrator, either manually or through a system-initiated script.

Zero or more dev_t entries (major and minor numbers) in /dev may be created by the device drivers each time that a device_t is created as part of the autoconfiguration process. Most device drivers create a single /dev entry, but network device drivers do not create any entries, whereas disk devices may create dozens. Additional entries may appear in /dev as the result of cloning devices. For example, a cloning device such as a pseudo-terminal creates a new device each time that it is opened.


7.2. The GEOM Layer

The GEOM layer provides a modular transformation framework for disk-I/O requests. This framework supports an infrastructure in which classes can do nearly arbitrary transformations on disk-I/O requests on their path from the upper kernel to the device drivers and back. GEOM can support both automatic data-directed configuration and manually or script-directed configuration. Transformations in GEOM include the following:

  • Simple base and bounds calculations needed for disk partitioning

  • Aggregation of disks to provide a RAID, mirrored, or stripped logical volume

  • A cryptographically protected logical volume

  • Collection of I/O statistics

  • I/O optimization such as disk sorting

Unlike many of its predecessors, GEOM is both extensible and topologically agnostic.

Terminology and Topology Rules

GEOM is object oriented and consequently borrows much context and semantics from the object-oriented terminology. A transformation is the concept of a particular way to modify I/O requests. Examples include partitioning a disk, mirroring two or more disks, and operating several disks together in a RAID.

A class implements one particular transformation. Examples of classes are a master boot record (MBR) disk partition, a BSD disk label, and a RAID array.

An instance of a class is called a geom. In a typical FreeBSD system, there will be one geom of class MBR for each disk. The MBR subdivides a disk into up to four pieces. There will also be one geom of class BSD for each slice with a BSD disk label.

A provider is the front gate at which a geom offers service. A typical provider is a logical disk, for example /dev/da0s1. All providers have three main properties: name, media size, and sector size.

A consumer is the backdoor through which a geom connects to another geom provider and through which I/O requests are sent. For example, an MBR label will typically be a consumer of a disk and a provider of disk slices.

The topological relationship between these entities are as follows:

  • A class has zero or more geom instances.

  • A geom is derived from exactly one class.

  • A geom has zero or more consumers.

  • A geom has zero or more providers.

  • A consumer can be attached to only one provider.

  • A provider can have multiple consumers attached.

  • The GEOM structure may not have loops; it must be an acyclic directed graph.

All geoms have a rank number assigned that detects and prevents loops in the acyclic directed graph. This rank number is assigned as follows:

  • A geom with no attached consumers has rank of one.

  • A geom with attached consumers has a rank one higher than the highest rank of the geoms of the providers to which its consumers are attached.

Figure 7.3 shows a sample GEOM configuration. At the bottom is a geom that communicates with the CAM layer and produces the da0 disk. It has two consumers. On the right is the DEVFS filesystem that exports the complete disk image as /dev/da0. On the left is stacked an MBR geom that interprets the MBR label found in the first sector of the disk to produce the two slices da0s1 and da0s2. Both of these slices have DEVFS consumers that export them as /dev/da0sl and /dev/da0s2. The first of these two slices has a second consumer, a BSD label geom, that interprets the BSD label found near the beginning of the slice. The BSD label subdivides the slice into up to eight (possibly overlapping) partitions da0sla through da0slh. All the defined partitions have DEVFS consumers that export them as /dev/da0sla through /dev/da0s1h. When one of these partitions is mounted, the filesystem that has mounted it also becomes a consumer of that partition.

Figure 7.3. A sample GEOM configuration.

Changing Topology

The basic operations are attach, which attaches a consumer to a provider, and detach, which breaks the bond. Several more complex operations are available to simplify automatic configuration.

Tasting is a process that happens whenever a new class or new provider is created. It provides the class a chance to automatically configure an instance on providers that it recognize as its own. A typical example is the MBR disk-partition class that will look for the MBR label in the first sector and if found and valid will instantiate a gcom to multiplex according to the contents of the MBR. Exactly what a class does to recognize if it should accept the offered provider is not defined by GEOM, but the sensible set of options are:

  • Examine specific data structures on the disk.

  • Examine properties like sector size or media size for the provider.

  • Examine the rank number of the provider's geom.

  • Examine the method name of the provider's geom.

A new class will be offered to all existing providers and a new provider will be offered to all classes.

Configure is the process where the administrator issues instructions for a particular class to instantiate itself. There are multiple ways to express intent. For example, a BSD label module can be specified with a level of override forcing a BSD disk-label geom to attach to a provider that was not found palatable during the taste operation. A configure operation is typically needed when first labelling a disk.

Orphaning is the process by which a provider is removed while it potentially is still being used. When a geom orphans a provider, all future I/O requests will bounce on the provider with an error code set by the geom. All consumers attached to the provider will receive notification about the orphaning and are expected to act appropriately. A geom that came into existence as a result of a normal taste operation should self-destruct unless it has a way to keep functioning without the orphaned provider. Single point of operation geoms, like those interpreting a disk label, should self-destruct. Geoms with redundant points of operation such as those supporting a RAID or a mirror will be able to continue as long as they do not lose quorum.

An orphaned provider may not result in an immediate change in the topology. Any attached consumers are still attached. Any opened paths are still open. Any outstanding I/O requests are still outstanding. A typical scenario is

  • A device driver detects a disk has departed and orphans the provider for it.

  • The geoms on top of the disk receive the orphaning event and orphan all their providers. Providers that are not in use will typically self-destruct immediately.

    This process continues in a recursive fashion until all relevant pieces of the tree have responded to the event.

  • Eventually the traversal stops when it reaches the device geom at the top of the tree. The geom will refuse to accept any new requests by returning an error. It will sleep until all outstanding I/O requests have been returned (usually as errors). It will then explicitly close, detach, and destroy its geom.

  • When all the geoms above the provider have disappeared, the provider will detach and destroy its geom. This process percolates all the way down through the tree until the cleanup is complete.

While this approach seems byzantine, it does provide the maximum flexibility and robustness in handling disappearing devices. By ensuring that the tree does not unravel until all the outstanding I/O requests have returned ensures that no applications will be left hanging because a piece of hardware has disappeared.

Spoiling is a special case of orphaning used to protect against stale metadata. It is probably easiest to understand spoiling by going through an example. Consider the configuration shown in Figure 7.3 that has disk da0 above, which is an MBR geom that provides da0s1 and da0s2. On top of da0s1, a BSD geom provides da0sla through da0s1h. Both the MBR and BSD geoms have autoconfigured based on data structures on the disk media. Now consider the case where da0 is opened for writing and the MBR is modified or overwritten. The MBR geom now would be operating on stale metadata unless some notification system can inform it otherwise. To avoid stale metadata, the opening of da0 for writing causes all attached consumers to be notified resulting in the eventual self-destruction of the MBR and BSD geoms. When da0 is closed, it will be offered for tasting again, and if the data structures for MBR and BSD are still there, new geoms will instantiate themselves.

To avoid the havoc of changing a disk label for an active filesystem, changing the size of open geoms can be done only with their cooperation. If any of the paths through the MBR or BSD geoms were open (for example, as a mounted filesystem), they would have propagated an exclusive-open flag downward, rendering it impossible to open da0 for writing. Conversely, the exclusive-open flag requested when opening da0 to rewrite the MBR would render it impossible to open a path through the MBR geom until da0 is closed. Spoiling only happens when the write count goes from zero to non-zero, and the tasting only happens when the write count goes from non-zero to zero.

Insert is an operation that allows a new geom to be instantiated between an existing consumer and provider. Delete is an operation that allows a geom to be removed from between an existing consumer and provider. These capabilities can be used to move an active filesystem. For example, we could insert a mirror module into the GEOM stack pictured in Figure 7.3 as shown in Figure 7.4 (on page 274). The mirror operates on da0s1 and dalsl between the BSD label consumer and its MBR label provider da0s1. The mirror is initially configured with da0s1 as its only copy and consequently is transparent to the I/O requests on the path. Next we ask it to mirror da0s1 to dalsl. When the mirror copy is complete, we drop the mirror copy on da0s1. Finally we delete the mirror geom from the path instructing the BSD label consumer to consume from dalsl. The result is that we moved a mounted file system from one disk to another while it was being used.

Figure 7.4. Using a mirror module to copy an active filesystem.

Operation

The GEOM system needs to be able to operate in a multiprocessor kernel. The usual method for ensuring proper operation is to use mutex locks on all the data structures. Because of the large size and complexity of the code and data structures implementing the GEOM classes, GEOM uses a single-threading approach rather than traditional mutex locking to ensure data structure consistency. GEOM uses two threads to operate its stack: a g_down thread to process requests moving from the consumers at the top to the producers at the bottom and a g_up thread to process requests moving from the producers at the bottom to the consumers at the top.

Requests entering the GEOM layer at the top are queued awaiting the g_down thread. The g_down thread pops each request from the queue, moves it down through the stack, and out through the producer. Similarly, results coming back from the producers are queued awaiting the g_up thread. The g_up thread pops each request from the queue, moves it up through the stack, and sends it back out to the consumer.

Because there is only ever a single thread running up and down in the stack, the only locking that is needed is on the few data structures that coordinate between the upward and downward paths. There are two rules required to make this single-thread method work effectively:

  1. A geom may never sleep. If a geom ever puts the g_up or g_down thread to sleep, the entire I/O system will grind to a halt until the geom reawakens. The GEOM framework checks that its worker threads never sleep, panicking if they attempt to do so.

  2. No geom can compute excessively. If a geom computes excessively, pending requests or results may be unacceptably delayed. There are some geoms, such as the one that provides cryptographic protection for filesystems, that are compute intensive. These compute-intensive geoms must provide their own threads. When the g_up or g_down thread enters a compute-intensive geom, it will simply enqueue the request, schedule the geom's own worker thread, and proceed on to process the next request in its queue. When scheduled, the compute-intensive geom's thread will do the needed work and then enqueue the result for the g_up or g_down thread to finish pushing the request through the stack.

The set of commands that may be passed through the GEOM stack are read, write, and delete. The read and write commands have the expected semantics. Delete specifies that a certain range of data are no longer used and that it can be erased or freed. Technologies like flash-adaptation layers can arrange to erase the relevant blocks before they can be reassigned and cryptographic devices may fill random bits into the range to reduce the amount of data available for attack. A delete request has no assurance that the data really will be erased or made unavailable unless guaranteed by specific geoms in the graph. If a secure-delete semantic is required, a geom should be pushed that converts a delete request into a sequence of write requests.

Topological Flexibility

GEOM is both extensible and topologically agnostic. The extensibility of GEOM makes it easy to write a new class of transformation. For example, if you needed to mount an IBM MVS disk, a class recognizing and configuring its Volume Table of Contents information would be trivial.

In a departure from many previous volume managers, GEOM is topologically agnostic. Most volume-management implementations have strict notions of how classes can fit together, but often only a single fixed hierarchy is provided. Figure 7.5 (on page 275) shows a typical hierarchy. It requires that the disks first be divided into partitions, and then the partitions can be grouped into mirrors, which are exported as volumes.

Figure 7.5. Fixed class hierarchy.

With fixed hierarchies, it is impossible to express intent efficiently. In the fixed hierarchy of Figure 7.5, it is impossible to mirror two physical disks and then partition the mirror into slices as is done in Figure 7.6. Instead one is forced to make slices on the physical volumes then create mirrors for each of the corresponding slices, resulting in a more complex configuration. Being topologically agnostic means that different orderings of classes are treated no differently than existing orderings. GEOM does not care in which order things are done. The only restriction is that cycles in the graph are not allowed.

Figure 7.6. Flexible class hierarchy.


7.3. The CAM Layer

To reduce the complexity of the individual disk drivers, much of the complexity of handling a modern controller has been abstracted out to a separate CAM layer that sits below the GEOM layer and above the device-driver level. The CAM layer handles the device-independent tasks of resource allocation and command routing. These tasks include the tracking of requests and notifications between the controller and its clients. They also include the routing of requests across the many I/O busses to get the request to the correct controller.

The CAM layer leaves to the device driver the device-specific operations such as the setup and teardown of the DMA (Direct Memory Access) maps needed to do the I/O. Some device drivers can still get complex. For example, the Fibre Channel device driver has a lot of code to handle operations, such as asynchronous topology changes as drives are removed and attached. A driver responds to a CAM request by converting the virtual address to store the data to the appropriate physical address. It then marshals the device-independent parameters like IO request, physical address to store the data, and transfer length into a firmware-specific format and executes the command. When the I/O completes, the driver returns the results back to the CAM layer.

In addition to disks, the CAM layer manages any other storage device that might be connected to the system such as tape drives and flash memory dongles. For other character devices such as keyboard and mice, CAM will not be involved. A USB keyboard will be attached directly to the keyboard driver.

SCSI Subsystem

The CAM SCSI subsystem provides a uniform and modular system for the implementation of drivers to control various SCSI devices and to use different SCSI host adapters through host-adapter drivers. The CAM system is made up of three layers:

  1. The CAM peripheral layer that provides open, close, strategy, attach, and detach operations for the supported SCSI devices. CAM supported devices includes direct access (da)—disk drives, cdrom (cd)—CD-ROM drives, sequential access (sa)—tape drives, and changer (ch)—jukeboxes.

  2. The CAM transport layer that builds, executes, and interprets results of SCSI commands. CAM starts by building a generic I/O command using a CCB (CAM Control Block). The CCB contains a CDB (Command Descriptor Block) composed of a 6-16 byte SCSI command. For example, the command "READ_10, block_offset, count" gets back a status of success or various error codes. If there is an error, the drive may also include sense data to give more information about the cause of the error.

  3. The CAM SIM (Software Interface Module) or HBA (Host Bus Adapter) interface layer provides bus routing to devices. Its job is to allocate a path to the requested device, send a CCB action request to the device, and then collect notification of the I/O completion from the device.

The operation of the CAM layer is most easily understood by tracing an I/O request through it.

The Path of an I/O Request Through the CAM Subsystem

The path of a request through the CAM I/O subsystem is shown in Figure 7.7 (on page 278). In the FreeBSD framework, the filesystem sees a single contiguous disk. I/O requests are based on block numbers within this idealized disk. In Figure 7.7, the filesystem determines a set of blocks on which it wants to do I/O, and it passes this request down to the GEOM layer by calling the strategy() routine.

Figure 7.7. The path of an I/O request through the CAM subsystem.

The GEOM layer takes the request and determines the disk to which the request should be sent. In this example the request is on a da SCSI disk. Sometimes a request may span several disks. When this happens, the GEOM layer breaks up the original request into a set of separate I/O requests for each of the disks on which the original request resides. Each of these new requests are passed down to the CAM layer by calling the appropriate strategy( ) routine for the associated disk (the dastrategy( ) routine in Figure 7.7).

The CAM dastrategy( ) routine gets the request and calls bioq_disksort( ), which puts the request on the disk queue of the specified SCSI disk. The dastrategy( ) routine finishes by calling the xpt_schedule( ) function.

The xpt_schedule( ) function allocates and constructs a CCB (CAM Control Block) to describe the operation that needs to be done. If the disk supports tagged queueing, an unused tag is allocated, if it is available. If tagged queueing is not supported or a tag is not available, the request is left on the queue of requests pending for the disk. If the disk is ready to accept a new command, the xpt_schedule( ) routine calls the drive start routine set up for it (dastart( ) in this example).

The dastart( ) routine takes the first request off the disk's queue and begins to service it using the CCB that was constructed by dastrategy( ). Because the command is destined for a SCSI disk, dastart( ) needs to build a SCSI READ_10 command based on the information in the CCB. The resulting SCSI command that includes a READ_10 header, a pointer to the virtual address that references the data to be transferred, and a transfer length is placed in the CCB and given the type XPT_SCSI_IO. The dastart( ) routine then calls the xpt_action( ) routine to determine the bus and controller (adaptor) to which the command should be sent.

The xpt_action( ) routine returns a pointer to a cam_path structure that describes the controller to be used and has a pointer to the controller's action routine. In this example we are using the Adaptec SCSI controller whose action routine is ahc_action(). The xpt_action() routine queues the CCB with its cam_path and schedules it to be processed.

The request is processed by calling the controller specific action routine, ahc_action(). The ahc_action() routine gets the CCB and converts its generic SCSI command into a hardware-specific SCB (SCSI Control Block) to handle the command. The SCB is filled out from information in the CCB. It is also filled out with any hardware-specific information and a DMA request descriptor is set up. The SCB is then passed to the driver firmware to be executed. Having completed its task, the CAM layer returns back to the caller of dastrategy().

The controller fulfills the request and DMAs the data to or from the location given in the SCB. When done, a completion interrupt arrives from the controller. The interrupt causes the ahc_done() routine to be run. The ahc_done() routine updates the CCB associated with the completed SCB from information in the SCB (command completion status or sense information if there was an error). It then frees the previously allocated DMA resources and the completed SCB and passes the completed CCB back to the CAM layer by calling xpt_done().

The xpt_done() routine inserts the associated CCB into the completion notification queue and posts a software interrupt request for camisr(), the CAM interrupt service routine. When camisr() runs, it removes the CCB from the completion notification queue and calls the specified completion function which maps to dadone() in this example.

The dadone() routine will call the biodone() routine, which notifies the GEOM layer that one of its I/O requests has finished. The GEOM layer aggregates all the separate disk I/O requests together. When the last I/O operation finishes, it updates the original I/O request passed to it by the filesystem to reflect the result (either successful completion or details on any errors that occurred). The filesystem is then notified of the final result by calling the biodone() routine.


7.4. The ATA Layer

To reduce the complexity of the individual disk drivers, much of the code common to supporting ATA drives has been abstracted out to a separate ATA layer that sits below the GEOM layer and above the device-driver level. The ATA layer handles the device-independent tasks of the tracking of request and notification between the controller and its clients. The ATA layer also handles the management of DMA mapping for the controllers that support it.

Device-specific operations are left to the device driver. The device driver responds to a request from ATA by marshaling the device-independent parameters like IO request, virtual address to store the data, and transfer length into a firmware-specific format, and then executing the command. When the I/O completes, the driver returns the results back to the ATA layer.

In addition to disks, the ATA layer manages any other storage device that might be connected to the system such as tape drives and CD-ROMs. The operation of the ATA layer is most easily understood by tracing an I/O request through it.

The Path of an I/O Request Through the ATA Subsystem

The path of a request through the ATA I/O subsystem is shown in Figure 7.8. As described in the previous section, the filesystem determines a set of blocks within its logical disk on which it wants to do I/O and passes this request down to the GEOM layer by calling the strategy() routine.

Figure 7.8. The path of an I/O request through the ATA subsystem.

The GEOM layer takes the request and determines the disk to which the request should be sent. In this example the request is on an ad ATA disk. Sometimes a request may span several disks. When this happens, the GEOM layer breaks up the original request into a set of separate I/O requests for each of the disks on which the original request resides. Each of these new requests is passed down to the ATA layer by calling the appropriate strategy() routine for the disk (see the adstrategy( ) routine in Figure 7.8).

The ATA adstrategy() routine gets the request and calls bioq_disksort(), which puts the request on the disk queue of the specified ATA disk. The adstrategy() routine finishes by calling the ata_start() function. The ata_start() routine is responsible for managing the channel queue. The channel queue is used to merge requests from the master and slave drives connected to the channel, since only one request can be active at a time. The ata_start() routine will alternate transfers between the two drives if both have outstanding requests. Once tag-queueing support is added, it will be possible to have outstanding requests on both drives. If the disk is not already busy, the ata_start() takes the next transfer request from the queue and invokes ata_transaction().

The ata_transaction() routine handles the low-level details of setting up a DMA mapping if the controller supports it. When the disk is ready to accept a new command, the ata_transaction() routine calls the drive-start routine set up for it by the adstrategy() routine (ad_start( ) in this example). The ad_start( ) routine takes the request off the channel queue and passes it to the driver firmware to be executed. Having completed its task, the ATA layer returns back to the caller of adstrategy ( ).

The controller fulfills the request. When done, a completion interrupt arrives from the controller. The interrupt causes the ata_interrupt( ) routine to be run. The ata_interrupt( ) routine updates the request with the command completion status or sense information if there was an error. It then frees any previously allocated DMA resources, releases the channel for new work, and passes the completed request back to the ATA layer by calling ata_finish( ). The ata_finish( ) routine puts the completed I/O request on an ATA work queue to be run by the g_up thread and returns from the interrupt.

When the g_up thread runs the ATA work queue, the completed I/O request is handled by ata_completed( ). The ata_completed( ) routine removes the request from its work queue, makes note of any errors that have occurred, and calls the specified completion function that maps to ad_done( ) in this example.

The GEOM layer aggregates all the separate disk I/O requests together. When the last I/O operation finishes, it updates the original I/O request passed to it by the filesystem to reflect the result (either successful completion or details on any errors that occurred). The filesystem is then notified of the final result by calling the biodone( ) routine.


7.5. Device Configuration

Autoconfiguration is the procedure carried out by the system to recognize and enable the hardware devices present in a system. Autoconfiguration works by systematically probing the possible I/O busses on the machine. For each I/O bus that is found, each type of device attached to it is interpreted, and, depending on this type, the necessary actions are taken to initialize and configure the device.

The first FreeBSD implementation of autoconfiguration was derived from the original 4.2BSD code with the addition of many special-case hacks. The 4.4BSD release introduced a new more machine-independent configuration system that was considered for FreeBSD but was ultimately rejected in favor of the newbus scheme used today. Newbus first appeared in FreeBSD 3.0 to support the Alpha architecture. It was brought over to the PC platform for FreeBSD 4.0. Newbus includes machine-independent routines and data structures for use by machine-dependent layers, and provides a framework for dynamic allocation of data structures for each device.

A key design goal of the newbus system was to expose a stable application binary interface (ABI) to driver writers. A stable ABI is especially important for externally or vendor-maintained loadable kernel modules because their source code is often not available to recompile if the interface is changed.

To help achieve ABI stability, the newbus device and devclass structures are hidden from the rest of the kernel with a simple function-call-based API to access their contents. If the structures were passed to the device driver directly, any change to the structure would require that all the drivers to which it is passed be recompiled. Changes to these newbus data structures do not require a recompilation of all the drivers. Only the access functions to the data structures need to be recompiled.

Unfortunately the goal of building a stable documented kernel ABI for driver writers has not yet been done outside of newbus. For example, the uio structure is still passed in to most drivers. Whenever a field is added, changed, or dropped from the uio structure, all the drivers must be recompiled.

Some hardware devices, such as the interface to the console terminal, are required for system operation. Other devices, however, may not be needed, and their inclusion in the system may needlessly waste system resources. Devices that might be present in different numbers, at different addresses, or in different combinations are difficult to configure in advance. However, the system must support them if they are present and must fail gracefully if they are not present. To address these problems, FreeBSD supports both a static configuration procedure that is done when a bootable system image is created and a dynamic loading capability that allows kernel drivers and modules to be added to a running system as needed. Thus, the statically configured kernel can be small with just enough capability to get the system up and running. Once running, additional functionality can be added as needed.

Allowing code to be loaded dynamically into the kernel raises many security problems. Code running outside the kernel is limited in the damage that it can do because it does not run in privileged mode and cannot directly access the hardware. The kernel runs with full privilege and access to the hardware. Thus, if it loads a module that contains malicious code, it can inflict wide-ranging damage within the system. Kernels can be loaded across the network from a central server. If the kernel allows dynamic loading of modules, they too could come across the network, so there are numerous added points for malfeasance.

An important consideration in deciding whether to enable dynamic loading of kernel modules is to develop a scheme to verify the source of and lack of corruption in any code before that code is permitted to be loaded and used. A group of vendors have formed the Trusted Computing Group (TCG) to specify a hardware module called a Trusted Platform Module (TPM) that keeps a running SHA-1 hash of the software installed on the system to detect the loading of bad programs or modules. It is implemented as a microcontroller-based device similar to a smart card that is attached to the motherboard [TCG, 2003]. Other groups are doing work to limit the potential harm of kernel modules by running them with page protections that limit their access to the rest of the kernel [Chiueh et al., 2004]. The drawback to disabling dynamic loading is that any hardware that is not included in the kernel configuration file will be unavailable for use.

The initial kernel configuration is done by the /usr/sbin/config program. A configuration file is created by the system administrator that contains a list of drivers and kernel options. Historically, the configuration file defined both the set of hardware devices that might be present on a machine and the location where each device might be found. In FreeBSD 5.2 most of the actual hardware devices are discovered dynamically as the various bus drivers probe and attach. The location of legacy devices for non-plug-and-play (non-self-identifying) busses are given in a /boot/device.hints file that is loaded with the kernel. The other use for hints is to hardwire a unit number to a location. Currently only CAM can hardwire unit numbers, although hard wiring could be implemented for any bus. The configuration procedure generates many files that define the initial kernel configuration. These files control the kernel compilation.

The autoconfiguration phase is done first during system initialization to identify the set of devices that are present on a machine. In general, autoconfiguration recurses through a tree of device interconnections, such as busses and controllers to which other devices attach. For example, a system might be configured with two SCSI host adapters (controllers) and four disk drives that are connected in any of the configurations shown in Figure 7.9. Autoconfiguration works in one of two ways at each level in the tree:

  1. Identifying each possible location at which a device might be present and checking to see what type of device (if any) is there.

  2. Probing for devices at each of the possible locations where the device might be attached.

Figure 7.9. Alternative drive configurations.

The first approach of identifying predefined locations for devices is needed for older busses like ISA that were not designed to support autoconfiguration. The second mechanism of probing for devices can be used only when a fixed set of locations are possible and when devices at those locations are self-identifying, such as devices connected to a SCSI or PCI bus.

Devices that are recognized during the autoconfiguration phase are attached and made available for use. The attach function for a device initializes and allocates resources for the device. The attach function for a bus or controller must probe for devices that might be attached at that location. If the attach function fails, the hardware was found but is nonfunctional, which results in a console message being printed. Devices that are present but not recognized may be configured once the system is running and has loaded other kernel modules. The attach function for busses is allowed to reserve resources for devices that are detected on the bus but for which no device driver is currently loaded in the system.

This scheme permits device drivers to allocate system resources for only those devices that are present in a running system. It allows the physical device topology to be changed without requiring the system load image to be regenerated. It also prevents crashes resulting from attempts to access a nonexistent device. In the remainder of this section, we consider the autoconfiguration facilities from the perspective of the device-driver writer. We examine the device-driver support required to identify hardware devices that are present on a machine and the steps needed to attach a device once its presence has been noted.

Device Identification

To participate in autoconfiguration, a device driver must register the set of functions shown in Table 7.1. Devices are an abstract concept in FreeBSD. In addition to the traditional disks, tapes, keyboards, terminal lines, and so on, FreeBSD will have devices that operate all the pieces that make up the I/O infrastructure such as the SCSI bus controller, the bridge controller to the PCI bus, and the bridge controller to the ISA bus. The top-level device is the root of the I/O system and is referred to as root0. On a uniprocessor system, root0 logically resides at the I/O pins of the CPU. On a multiprocessor system, root0 is logically connected to the I/O pins of each of the CPUs. The root0 device is handcrafted at boot time for each architecture supported by FreeBSD.

Table 7.1. Functions defined for autoconfiguration.

Function

Description

device_probe

probe device presence

device_identify

add new device to bus

device_attach

attach a device

device_detach

detach a device

device_shutdown

system about to shut down

device_suspend

suspend requested

device_resume

resume has occurred

Autoconfiguration begins with a request to the root0 bus to configure all its children. When a bus is asked to configure its children, it calls the device_identify( ) routine of each of its possible device drivers. The result is a set of children that have been added to the bus either by the bus itself or by the device_identify( ) routines of one of its drivers. Next the device_probe( ) routine of each of the children is called. The device_probe( ) routine that bids the highest for the device will then have its device_attach( ) routine called. The result is a set of devices corresponding to each of the busses that are directly accessible to root0. Each of these new devices is then given the opportunity to probe for or identify devices below them. The identification process continues until the topology of the I/O system has been determined.

Modern busses can directly identify the things that are connected to them. Older busses such as ISA use the device_identify( ) routine to bring in devices that are found only because of hints.

As an example of the device hierarchy, the device controlling the PCI bus may probe for a SCSI controller, which in turn will probe for possible targets that might be attached, such as disk and tape drives. The newbus autoconfiguration mechanism provides much flexibility, allowing a controller to determine the appropriate way in which to probe for additional devices attached to the controller.

As autoconfiguration proceeds, a device-driver device_probe( ) routine is called for each device that is found. The system passes to the device_probe( ) routine a description of the device's location and possibly other details such as I/O register location, memory location, and interrupt vectors. The device_probe( ) routine usually just checks to see if it recognizes the hardware.

It is possible that there is more than one driver that can operate a device. Here, each matching driver returns a priority that shows how well they match the hardware. Success codes are values less than or equal to zero with the highest value representing the best match. Failure codes are represented by positive values using the usual kernel error codes.

If a driver returns a success code that is less than zero, it must not assume that it will be the same driver whose device_attach( ) routine will be called. In particular, it must not assume that any values stored in the device local-storage area will be available for its device_attach( ) routine. Any resources allocated during the probe must be released and reallocated if its device_attach( ) routine is called. By returning a success code of zero, a driver can assume that it will be the one attached. However, well-written drivers will not have their device_attach( ) routine use the device local-storage area because they may one day have their return value downgraded to a value less than zero. Typically the resources used by a device are identified by the bus (parent device), and it is the bus that prints them out when the devices are probing.

Once the device_probe( ) routine has had the opportunity to identify the device and select the most appropriate driver to operate it, the device_attach( ) routine is called. Attaching a device is separated from probing so that drivers can be bid for devices. Probe and attach are also separate so that drivers can separate out the identification part of the configuration from the attaching part. Most device drivers use the device_attach( ) routine to initialize the hardware device and any software state. The device_attach( ) routine is also responsible for either creating the dev_t entries (for disks and character devices) or for network devices, registering the device with the networking system.

Devices that represent pieces of hardware such as a SCSI controller will respond to verify that the device is present and to set or at least identify the device's interrupt vector. For disk devices, the device_attach( ) routine may make the drive available to higher levels of the kernel such as GEOM. GEOM will let its classes taste the disk drive to identify its geometry and possibly initialize the partition table that defines the placement of filesystems on the drive.

Autoconfiguration Data Structures

The newbus autoconfiguration system in FreeBSD includes machine-independent data structures and support routines. The data structures allow machine- and bus-dependent information to be stored in a general way and allow the autoconfiguration process to be driven by the configuration data, rather than by compiled-in rules. The /usr/sbin/config program constructs many of the tables from information in the kernel-configuration file and from a machine-description file. The /usr/sbin/config program is thus data-driven as well and contains no machine-dependent code.

Figure 7.10 shows the data structures used by autoconfiguration. The basic building block is the device structure. Each piece of the I/O hierarchy will have its own device structure. The name and description fields identify the piece of hardware represented by the device structure. In Figure 7.10, the name of the device is pcil. Device names are globally unique. There can be only one pcil device in the system. Knowing its name is enough to find it unlike filesystems where there can be many files with the same name in different paths. This name space is related by convention to the name space that /dev entries have, but such a relationship is not required.

Figure 7.10. Autoconfiguration data structures for pcil.

[View full size image]

Each device is a member of a device class represented by a devclass structure that has two important roles. The first role of the devclass structure is to keep track of a list of drivers for devices in that class. Devices referenced from a devclass do not have to use the same driver. Each device structure references its best matching driver from the list available for the devclass. The list of candidate drivers is traversed, allowing every driver to probe each device that is identified as a member of the class. The best matching driver will be attached to the device. For example, the pci devclass contains a list of drivers suitable for probing against devices that may be plugged into a PCI bus. In Figure 7.10 there are drivers to match pcm (sound cards) and atapci (PCI-based ATA-disk controllers).

The second role of the devclass structure is to manage the mapping from a user-friendly device name such as pcil to its device structure. The name field in the devclass structure contains the root of a family of names—in this example, pci. The number following the root of the name—1 in this example—indexes into the array of pointers to device structures contained in the devclass. The name in the referenced device structure is the full name, pcil.

When a device structure first comes into existence, it will take the following steps:

1. The parent device determines the existence of a new child device typically by doing a bus scan. The new device is created as a child of the parent. In Figure 7.10, the autoconfiguration code would begin a scan of pcil and discover an ATA disk controller.

2. The parent device starts a probe-and-attach sequence for the new child device. The probe iterates through drivers in the parent device's devclass until a driver is found that claims the device (i.e., the probe succeeds). The device structure sets its driver field to point at the selected driver structure and increments the reference count in the selected driver. In Figure 7.10, the atapci driver matches the ATA disk controller, atadisk.

3. Once a usable driver is found, the new device is registered with the devclass of the same name as the driver. The registration is done by allocating the next available unit number and setting a pointer from the corresponding entry in the devclass's array of device-structure pointers back to the device. In Figure 7.10, the atapci driver was matched, so the device would be bound to the atapci devclass. The resulting device configuration is shown in Figure 7.11 (on page 288). The key observation is that two different devclasses are involved in this three-step process.

Figure 7.11. Autoconfiguration data structures for atapci0.

[View full size image]

The hierarchy of device structures is shown in Figure 7.12. Each device structure has a parent pointer and a list of children. In Figure 7.12, the pci device that manages the PCI bus is shown at the top and has as its only child the atapci device that operates ATA disks on the PCI bus. The atapci device has the pci device as its parent and has two children, one for each of the ATA disks that are attached. The devices representing the two drives have the atapci device as their parent. Because they are leaf nodes, they have no children.

Figure 7.12. Sample hierarchy of device structures.

To get a better idea of the I/O hierarchy, an annotated copy of the output of the /usr/sbin/devinfo program from the first author's test machine is shown in Figure 7.13 (on page 290). The output has been trimmed down from its original 250 lines to show just the branch from the root of the tree to the system's two ATA disks. The tree starts at root0, representing the I/O pins on the CPU. That leads to the high-speed bus that connects to the memory and the nexus0 (for example, northbridge) interconnect to the I/O bus. For machines on which ACPI is not available, there is still a legacy0 shim to help identify the I/O busses on the other side of the northbridge. One of these busses is the pcib0 (for example, south-bridge) connection to the PCI bus. The PCI bus is managed by the pci0 device, which as you can see from the figure, has many drivers available for the myriad of devices that may be connected to it. In this example, the one device that we show is the atapci0 device representing the PCI-based ATA disk controller. The final two devices shown in Figure 7.13 are atadisk0 and atadisk1 that manage the operation of the drives themselves.

Resource Management

As part of configuring and operating devices, the autoconfiguration code needs to manage hardware resources such as interrupt-request lines, I/O ports, and device memory. To aid device-driver writers in this task, newbus provides a framework for managing these resources. To participate in newbus bus-resource management, a bus device driver must register the set of functions shown in Table 7.2 (on page 291). Low-level devices such as those that operate individual disk drives do not have the global knowledge of resource utilization needed to allocate scarce systemwide resources such as interrupt-request lines. They may register a generic bypass routine for resources that they do not have the needed information to allocate. When called, the bypass routine simply calls the corresponding routine registered by their parent. The result is that the request will work its way up the device tree until it reaches a high enough level that it can be resolved.

Table 7.2. Functions defined for device resource allocation.

Function

Description

bus_alloc_resource

allocate a bus resource

bus_set_resource

set up a range for a resource

bus_activate_resource

activate allocated resource

bus_deactivate_resource

deactivate allocated resource

bus_release_resource

release ownership of a resource

bus_delete_resource

free a released resource

bus_get_resource

get the range for a resource

bus_get_resource_list

get a resource list

bus_probe_nomatch

call following a failed probe

bus_driver_added

new driver added to devclass

bus_add_child

attach an identified device

bus_child_detached

notice to parent of detached child

bus_setup_intr

initialize an interrupt

bus_config_intr

set interrupt trigger mode and polarity

bus_teardown_intr

disable an interrupt

bus_read_ivar

read an instance variable

bus_write_ivar

write an instance variable

bus_child_present

check for device still present

bus_print_child

print device description

Often a high-level node in the tree will not have enough information to know how much of a resource to allocate. So it will reserve a range of resources, leaving it to the lower-level nodes to allocate and activate the specific resources that they need from the reservation made by the higher-level node.

The actual management of the allocated resources is handled by the kernel resource manager that was described in Section 5.10. The usual allocate and free routines have been expanded to allow different levels in the tree to manage different parts of these two functions. Thus, allocation breaks into three steps:

1. Setting the range for the resource

2. Initial allocation of the resource

3. Activating the resource

Figure 7.13. Sample configuration output.
root0
    description: System root bus
    devclass: root, drivers: nexus
    children: nexus0

  nexus0
      devclass: nexus, drivers: acpi, legacy, npx
      children: npx0, legacy0

    legacy0
        description: legacy system
        devclass: legacy, drivers: eisa, isa, pcib
        children: eisa0, pcib0

      pcib0 /* southbridge */
          description: Intel 82443BX (440 BX) host to PCI bridge
          devclass: pcib, drivers: pci
          children: pci0
      pcibl
          description: PCI-PCI bridge
          devclass: pcib, drivers: pci
          children: pcil

        pci0
            description: PCI bus
            devclass: pci, drivers: agp, ahc, amr, asr, atapci,
                bfe, bge, ciss, csa, domino, dpt, eisab, emujoy,
                fxp, fixup_pci, hostb, ignore_pci, iir, ips,
                isab, mly, mode0, pcib, pcm, re, sio, uhci, xl
            children: agp0, pcibl, isab0, atapci0, ahc0, x10

          atapci0
              description: Intel 82371AB PIIX4 IDE controller
              devclass: atapci, drivers: atadisk, atapicd
              children: atadisk0, atadiskl
              class: mass storage, subclass: ATA
              I/O ports: 0xffa0-0xffaf

            atadisk0
                devclass: atadisk, drivers: none
                interrupt request lines: 0xe
                I/O ports: 0xlf0-0xlf7, 0x3f6

            atadiskl
                devclass: atadisk, drivers: none
                interrupt request lines: 0xf
                I/O ports: 0x170-0x177, 0x376

Similarly, resource freeing is done in three steps:

1. Deactivating the resource

2. Releasing ownership of the resource to the parent bus

3. Freeing it

It is common for a high-level part of the tree to allocate a resource and then have a low-level driver activate and use the resource that it was allocated. Some busses reserve space for their children that do not have drivers associated with them. Having the allocation and freeing broken up into three steps gives maximum flexibility in partitioning the allocation and freeing processes.

The bus_driver_added( ), bus_add_child( ), and bus_child_detached( ) functions allow the device to be aware of changes in the I/O hardware so that it can respond appropriately. The bus_driver_added( ) function is called by the system when a new driver is loaded. The driver is added to some devclass, and then all current devices in that devclass have bus_driver_added( called to allow them to possibly match any unclaimed devices using the new driver. The bus_add_child( ) function is used during the identify phase of configuring some busses. It allows a bus device to create and initialize a new child device (for example, setting values for instance variables). The bus_child_detached( ) function is called by a driver when it decides that its hardware is no longer present (for example, a cardbus card is removed). It calls bus_child_detached( on its parent to allow it to do a detach of the child.

The bus_probe_nomatch( ) routine gives the device a last-ditch possibility to take some action after autoconfiguration has failed. It may try to find a generic driver that can run the device in a degraded mode, or it may simply turn the device off. If it is unable to find a driver that can run the device, it notifies the devd daemon, a user-level process started when the system is booted. The devd daemon uses a table to locate and load the proper driver. The loading of kernel modules is described in Section 14.4.

The bus_read_ivar( ) and bus_write_ivar( ) routines manage a bus-specific set of instance variables of a child device. The intention is that each different type of bus defines a set of appropriate instance variables such as ports and interrupt-request lines for the ISA bus.


Exercises

7.1 Name the three layers between the filesystem and the disk. Briefly describe the purpose of each layer.

7.2 Describe the differences between the PCI and USB busses.

7.3 Why was the /dev filesystem added to FreeBSD 5.2?

7.4 Explain the difference between a dev_t and a device_t.

7.5 Give an example of a GEOM provider and a GEOM consumer.

7.6 What happens if two GEOM consumers try to operate at the same time?

7.7 Draw a sequence of pictures showing what happens to the GEOM configuration in Figure 7.3 when the disk becomes unavailable.

7.8 Name the three layers within CAM. Briefly describe the service that each of these layers provides.

7.9 Can the CAM layer handle the setup and tear down of the DMA maps for one of its device drivers? Why or why not?

7.10 What is the purpose of the /usr/sbin/config program?

7.11 Give two reasons why it is unsafe to allow a kernel to dynamically load code.

7.12 Why are device probing and attaching done as two separate steps?

7.13 Describe the purpose of the device structure.

7.14 Run the /usr/sbin/devinfo program on a FreeBSD machine and identify the hardware associated with each of the leaf nodes.

7.15 Name the three steps used for resource allocation and freeing. Why are these functions broken into three separate steps?

**7.16 All devices are currently attached once in a depth-first search. But some devices may offer services needed by devices higher up the tree. Describe devices that fall into this class and give a plan to build a multipass-attach approach into newbus to handle them.


References

ACPI, 2003. ACPI, "Advanced Configuration and Power Interface Specification," Web Reference, http://www.acpi.info/, November 2003.

ANSI, 2002. ANSI, "Common Access Method draft standard," X3T10, available from http://www.t10.org, January 2002.

ANSI, 2003. ANSI, "Fibre Channel draft standard," T11, available from http://www.t11.org, January 2003.

Arch, 2003. Arch, "PC Architecture," Web References, http://www.intel.com/design/chipsets/865PE/pix/865PE_schematic.gif, http://www.just2good.co.uk/chipset.htm, and http://www.gotogeek.com/mobopics/, November 2003.

Chiueh et al., 2004. T. Chiueh, P. Pardhan, & G. Venkitachalam, Intra-Address Space Protection Using Segmentation Hardware, http://www.ecsl.cs.sunysb.edu/palladium.html, March 2004.

TCG, 2003. TCG, Trusted Computing Group TPM Specification Version 1.2, http://www.trustedcomputinggroup.org/ and http://www.infineon.com/cgi/ecrm.dll/ecrm/scripts/prod_ov.jsp?oid=29049, October 2003.


Chapter 8. Local Filesystems

    Section 8.1. Hierarchical Filesystem Management

    Section 8.2. Structure of an Inode

    Section 8.3. Naming

    Section 8.4. Quotas

    Section 8.5. File Locking

    Section 8.6. Soft Updates

    Section 8.7. Filesystem Snapshots

    Section 8.8. The Local Filestore

    Section 8.9. The Berkeley Fast Filesystem

    Exercises

    References


8.1. Hierarchical Filesystem Management

The operations defined for local filesystems are divided into two parts. Common to all local filesystems are hierarchical naming, locking, quotas, attribute management, and protection. These features, which are independent of how data are stored, are provided by the UFS code described in the first seven sections of this chapter. The other part of the local filesystem, the filestore, is concerned with the organization and management of the data on the storage media. Storage is managed by the datastore filesystem operations described in the final two sections of this chapter.

The vnode operations defined for doing hierarchical filesystem operations are shown in Table 8.1 (on page 296). The most complex of these operations is that for doing a lookup. The filesystem-independent part of the lookup is described in Section 6.5. The algorithm used to lookup a pathname component in a directory is described in Section 8.3.

Table 8.1. Hierarchical filesystem operations.

Operation done

Operator names

pathname searching

lookup

name creation

create, mknod, link, symlink, mkdir

name change/deletion

rename, remove, rmdir

attribute manipulation

access, getattr, setattr

object interpretation

open, readdir, readlink, mmap, close

process control

advlock, ioctl, poll

object management

lock, unlock, inactive, reclaim

There are five operators for creating names. The operator used depends on the type of object being created. The create operator creates regular files and also is used by the networking code to create AF_LOCAL domain sockets. The link operator creates additional names for existing objects. The symlink operator creates a symbolic link (see Section 8.3 for a discussion of symbolic links). The mknod operator creates character special devices (for compatibility with other UNIX systems that still use them); it is also used to create fifos. The mkdir operator creates directories.

There are three operators for changing or deleting existing names. The rename operator deletes a name for an object in one location and creates a new name for the object in another location. The implementation of this operator is complex when the kernel is dealing with the movement of a directory from one part of the filesystem tree to another. The remove operator removes a name. If the removed name is the final reference to the object, the space associated with the underlying object is reclaimed. The remove operator operates on all object types except directories; they are removed using the rmdir operator.

Three operators are supplied for object attributes. The kernel retrieves attributes from an object using the getattr operator and stores them using the setattr operator. Access checks for a given user are provided by the access operator.

Five operators are provided for interpreting objects. The open and close operators have only peripheral use for regular files, but when they are used on special devices, they notify the appropriate device driver of device activation or shutdown. The readdir operator converts the filesystem-specific format of a directory to the standard list of directory entries expected by an application. Note that the interpretation of the contents of a directory is provided by the hierarchical filesystem-management layer; the filestore code considers a directory as just another object holding data. The readlink operator returns the contents of a symbolic link. As with directories, the filestore code considers a symbolic link as just another object holding data. The mmap operator prepares an object to be mapped into the address space of a process.

Three operators are provided to allow process control over objects. The poll operator allows a process to find out whether an object is ready to be read or written. The ioctl operator passes control requests to a special device. The advlock operator allows a process to acquire or release an advisory lock on an object. None of these operators modifies the object in the filestore. They are simply using the object for naming or directing the desired operation.

There are four operations for management of the objects. The inactive and reclaim operators were described in Section 6.6. The lock and unlock operators allow the callers of the vnode interface to provide hints to the code that implement operations on the underlying objects. Stateless filesystems such as NFS ignore these hints. Stateful filesystems, however, can use hints to avoid doing extra work. For example, an open system call requesting that a new file be created requires two steps. First, a lookup call is done to see if the file already exists. Before the lookup is started, a lock request is made on the directory being searched. While scanning through the directory checking for the name, the lookup code also identifies a location within the directory that contains enough space to hold the new name. If the lookup returns successfully (meaning that the name does not already exist), the open code verifies that the user has permission to create the file. If the caller is not eligible to create the new file, then they are expected to call unlock to release the lock that they acquired during the lookup. Otherwise, the create operation is called. If the filesystem is stateful and has been able to lock the directory, then it can simply create the name in the previously identified space, because it knows that no other processes will have had access to the directory. Once the name is created, an unlock request is made on the directory. If the filesystem is stateless, then it cannot lock the directory, so the create operator must rescan the directory to find space and to verify that the name has not been created since the lookup.


8.2. Structure of an Inode

To allow files to be allocated concurrently and to provide random access within files, FreeBSD uses the concept of an index node, or inode. The inode contains information about the contents of the file; see Figure 8.1 (on page 298). This information includes the following:

  • The type and access mode for the file

  • The file's owner and group-access identifiers

  • The time that the file was created, when it was most recently read and written, and when its inode was most recently updated by the system

  • The size of the file in bytes

  • The number of physical blocks used by the file (including blocks used to hold indirect pointers and extended attributes)

  • The number of directory entries that reference the file

  • The kernel and user setable flags that describe characteristics of the file

  • The generation number of the file (a randomly selected number assigned to the inode each time that the latter is allocated to a new file; the generation number is used by NFS to detect references to deleted files)

  • The blocksize of the data blocks referenced by the mode (typically the same as, but sometimes larger than, the filesystem blocksize)

  • The size of the extended attribute information

Figure 8.1. The structure of an inode.

Notably missing in the mode is the filename. Filenames are maintained in directories rather than in modes because a file may have many names, or links, and the name of a file can be large (up to 255 bytes in length). Directories are described in Section 8.3.

To create a new name for a file, the system increments the count of the number of names referring to that inode. Then the new name is entered in a directory, along with the number of the inode. Conversely, when a name is deleted, the entry is deleted from a directory, and the name count for the inode is then decremented. When the name count reaches zero, the system deallocates the inode by putting all the mode's blocks back on a list of free blocks.

The inode also contains an array of pointers to the blocks in the file. The system can convert from a logical block number to a physical sector number by indexing into the array using the logical block number. A null array entry shows that no block has been allocated and will cause a block of zeros to be returned on a read. On a write of such an entry, a new block is allocated, the array entry is updated with the new block number, and the data are written to the disk.

Inodes are fixed in size, and most files are small, so the array of pointers must be small for efficient use of space. The first 12 array entries are allocated in the inode itself. For typical filesystems, this implementation allows the first 96 or 192 Kbyte of data to be located directly via a simple indexed lookup.

For somewhat larger files, Figure 8.1 shows that the inodc contains a single indirect pointer that points to a single indirect block of pointers to data blocks. To find the 100th logical block of a file, the system first fetches the block identified by the indirect pointer and then indexes into the 88th block (100 minus 12 direct pointers) and fetches that data block.

For files that are bigger than a few megabytes, the single indirect block is eventually exhausted; these files must resort to using a double indirect block, which is a pointer to a block of pointers to pointers to data blocks. For files of multiple Gbyte, the system uses a triple indirect block, which contains three levels of pointer before reaching the data block.

Although indirect blocks appear to increase the number of disk accesses required to get a block of data, the overhead of the transfer is typically much lower. In Section 6.6, we discussed the management of the cache that holds recently used disk blocks. The first time that a block of indirect pointers is needed, it is brought into the cache. Further accesses to the indirect pointers find the block already resident in memory; thus, they require only a single disk access to get the data.

Changes to the Inode Format

Traditionally, the FreeBSD fast filesystem (which we shall refer to in this book as UFS1) [McKusick et al., 1984] and its derivatives have used 32-bit pointers to reference the blocks used by a file on the disk. The UFS1 filesystem was designed in the early 1980s when the largest disks were 330 Mbytes. There was debate at the time whether it was worth squandering 32 bits per block pointer rather than using the 24-bit block pointers of the filesystem that it replaced. Luckily the futurist view prevailed, and the design used 32-bit block pointers. Over the 20 years since it has been deployed, storage systems have grown to hold over a terabyte of data. Depending on the block size configuration, the 32-bit block pointers of UFS1 run out of space in the 1 to 4 terabyte range. While some stopgap measures can be used to extend the maximum-size storage systems supported by UFS1, by 2002 it became clear the only long-term solution was to use 64-bit block pointers. Thus, we decided to build a new filesystem, UFS2, that would use 64-bit block pointers.

We considered the alternatives between trying to make incremental changes to the existing UFS1 filesystem versus importing another existing filesystem such as XFS [Sweeney et al., 1996], or ReiserFS [Reiser, 2001]. We also considered writing a new filesystem from scratch so that we could take advantage of recent filesystem research and experience. We chose to extend the UFS1 filesystem because this approach allowed us to reuse most of the existing UFS1 code base. The benefits of this decision were that UFS2 was developed and deployed quickly, it became stable and reliable rapidly, and the same code base could be used to support both UFS1 and UFS2 filesystem formats. Over 90 percent of the code base is shared, so bug fixes and feature or performance enhancements usually apply to both filesystem formats.

The on-disk inodcs used by UFS1 are 128 bytes in size and have only two unused 32-bit fields. It would not be possible to convert to 64-bit block pointers without reducing the number of direct block pointers from 12 to 5. Doing so would dramatically increase the amount of wasted space, since only direct block pointers can reference fragments, so the only alternative is to increase the size of the on-disk mode to 256 bytes.

Once one is committed to changing to a new on-disk format for the modes, it is possible to include other inode-related changes that were not possible within the constraints of the old modes. While it was tempting to throw in everything that has ever been suggested over the last 20 years, we felt that it was best to limit the addition of new capabilities to those that were likely to have a clear benefit. Every new addition adds complexity that has a cost both in maintainability and performance. Obscure or little-used features may add conditional checks in frequently executed code paths such as read and write, slowing down the overall performance of the filesystem even if they are not used.

Extended Attributes

A major addition in UFS2 is support for extended attributes. Extended attributes are a piece of auxiliary data storage associated with an inode that can be used to store auxiliary data that is separate from the contents of the file. The idea is similar to the concept of data forks used in the Apple filesystem [Apple, 2003]. By integrating the extended attributes into the mode itself, it is possible to provide the same integrity guarantees as are made for the contents of the file itself. Specifically, the successful completion of an fsync system call ensures that the file data, the extended attributes, and all names and paths leading to the names of the file are in stable store.

The current implementation has space in the mode to store up to two blocks of extended attributes. The new UFS2 inode format had room for up to five additional 64-bit pointers. Thus, the number of extended attribute blocks could have been between one to five blocks. We chose to allocate two blocks to the extended attributes and to leave the other three as spares for future use. By having two, all the code had to be prepared to deal with an array of pointers, so if the number got expanded into the remaining spares in the future, the existing implementation will work without changes to the source code. By saving three spares, we provided a reasonable amount of space for future needs. And if the decision to allow only two blocks proves to be too little space, one or more of the spares can be used to expand the size of the extended attributes in the future. If vastly more extended attribute space is needed, a spare could be used as an indirect pointer to extended attribute data blocks.

Figure 8.2 shows the format used for the extended attributes. The first field of the header of each attribute is its length. Applications that do not understand the name space or name can simply skip over the unknown attribute by adding the length to their current position to get to the next attribute. Thus, many different applications can share the usage of the extended attribute space, even if they do not understand each other's data types.

Figure 8.2. Format of extended attributes. The header of each attribute has a 4-byte length, 1-byte name-space class, 1-byte content pad length, 1-byte name length, and name. The name is padded so that the contents start on an 8-byte boundary. The contents are padded to the size shown by the "content pad length" field. The size of the contents can be calculated by subtracting from the length the size of the header (including the name) and the content pad length.

The first of two initial uses for extended attributes is to support an access control list, generally referred to as an ACL. An ACL replaces the group permissions for a file with a more specific list of the users that are permitted to access the files. The ACL also includes a list of the permissions that each user is granted. These permissions include the traditional read, write, and execute permissions along with other properties such as the right to rename or delete the file [Rhodes, 2003].

Earlier implementations of ACLs were done with a single auxiliary file per filesystem that was indexed by the inode number and had a small fixed-sized area to store the ACL permissions. The small size was to keep the size of the auxiliary file reasonable, since it had to have space for every possible inode in the filesystem. There were two problems with this implementation. The fixed size of the space per inode to store the ACL information meant that it was not possible to give access to long lists of users. The second problem was that it was difficult to atomically commit changes to the ACL list for a file, since an update required that both the file inode and the ACL file be written to have the update take effect [Watson, 2000].

Both problems with the auxiliary file implementation of ACLs are fixed by storing the ACL information directly in the extended-attribute data area of the inode. Because of the large size of the extended attribute data area (a minimum of 8 Kbytes and typically 32 Kbytes), long lists of ACL information can be easily stored. Space used to store extended attribute information is proportional to the number of inodes with extended attributes and the size of the ACL lists that they use. Atomic update of the information is much easier, since writing the inode will update the inode attributes and the set of data that it references including the extended attributes in one disk operation. While it would be possible to update the old auxiliary file on every fsync system call done on the filesystem, the cost of doing so would be prohibitive. Here, the kernel knows if the extended attribute data block for an inode is dirty and can write just that data block during an fsync call on the inode.

The second use for extended attributes is for data labeling. Data labels provide permissions for a mandatory access control (MAC) framework enforced by the kernel. The kernel's MAC framework permits dynamically introduced system-security modules to modify system security functionality. This framework can be used to support a variety of new security services, including traditional labeled mandatory access control models. The framework provides a series of entry points that are called by code supporting various kernel services, especially with respects to access control points and object creation. The framework then calls out to security modules to offer them the opportunity to modify security behavior at those MAC entry points. Thus, the filesystem does not codify how the labels are used or enforced. It simply stores the labels associated with the inode and produces them when a security modules needs to query them to do a permission check [Watson, 2001; Watson et al., 2003].

We considered storing symbolic links in the extended attribute area but chose not to do so for four reasons:

  1. First, most symbolic links fit within the 120 bytes normally used to store the direct and indirect pointers and thus do not need a disk block to be allocated to hold them.

  2. If the symbolic link is large enough to require storage in a disk block, the time to access an extended storage block is the same as the time to access a regular data block.

  3. Since symbolic links rarely have any extended attributes, there would be no savings in storage, since a filesystem fragment would be needed whether it was stored in a regular data block or in an extended storage block.

  4. If it were stored in the extended storage area, it would take more time to traverse down the attribute list to find it.

New Filesystem Capabilities

Several other improvements were made when the enlarged inode format was created. We decided to get an early jump on the year 2038 problem when the 32-bit time fields overflow (specifically, Tue Jan 19 03:14:08 2038 GMT, which could be a really ugly way to usher in the first author's 84th birthday). We expanded the time fields (which hold seconds-since-1970) for access, modification, and inode-modification times from 32 bits to 64 bits. At plus or minus 136 billion years that should carry us from well before the universe was created until long after our sun has burned itself out. We left the nanoseconds fields for these times at 32 bits because we did not feel that added resolution was going to be useful in the foreseeable future. We considered expanding the time to only 48 bits. We chose to go to 64 bits, since 64 bits is a native size that can be easily manipulated with existing and likely future architectures. Using 48 bits would have required an extra unpacking or packing step each time the field was read or written. Also, going to 64 bits ensures enough bits for all likely measured time so it will not have to be enlarged.

We also added a new time field (also 64 bits) to hold the birth time (also commonly called the creation time) of the file. The birth time is set when the inode is first allocated and is not changed thereafter. It has been added to the structure returned by the stat system call so that applications can determine its value and so that archiving programs such as dump, tar, and pax can save this value along with the other file times. The birth time was added to a previously spare field in the stat system call structure so that the size of the structure did not change. Thus, old versions of programs that use the stat call continue to work.

To date, only the dump program has been changed to save the birth time value. This new version of dump, which can dump both UFS1 and UFS2 filesystems, creates a new dump format that is not readable by older versions of restore. The updated version of restore can identify and restore from both old and new dump formats. The birth times are only available and setable from the new dump format.

The utimes system call sets the access and modification times of a file to a specified set of values. It is used primarily by archive retrieval programs to set a newly extracted file's times back to those associated with the file's times in the archive. With the addition of birth time, we added a new system call that allows the setting of access, modification, and birth times. However, we realized that many existing applications will not be changed to use the new utimes system call. The result will be that the files that they retrieved from archives will have a newer birth time than access or modification times.

To provide a sensible birth time for applications that are unaware of the birth time attribute, we changed the semantics of the utimes system call so that if the birth time was newer than the value of the modification time that it was setting, it sets the birth time to the same time as the modification time. An application that is aware of the birth time attribute can set both the birth time and the modification time by doing two calls to utimes. First, it calls utimes with a modification time equal to the saved birth time, and then it calls utimes a second time with a modification time equal to the (presumably newer) saved modification time. For filesystems that do not store birth times, the second call will overwrite the first call resulting in the same values for access and modification times as they would have previously gotten. For filesystems that support birth time, it will be properly set. Most happily for the application writers, they will not have to conditionally compile the name of utimes for BSD and non-BSD systems. They just write their applications to call the standard interface twice knowing that the right thing will happen on all systems and filesystems. For those applications that value speed of execution over portability can use the new version of the utimes system call that allows all time values to be set with one call.

File Flags

FreeBSD has two system calls, chflags and fchflags, that set the 32-bit user-flags word in the inode. The flags are included in the stat structure so that they can be inspected.

The owner of the file or the superuser can set the low 16 bits. Currently, there are flags defined to mark a file as append-only, immutable, and not needing to be dumped. An immutable file may not be changed, moved, or deleted. An append-only file is immutable, except data may be appended to it. The user append-only and immutable flags may be changed by the owner of the file or the superuser.

Only the superuser can set the high 16 bits. Currently, there are flags defined to mark a file as append-only and immutable. Once set, the append-only and immutable flags in the top 16 bits cannot be cleared when the system is secure.

The kernel runs with four different levels of security. Any superuser process can raise the security level, but only the init process can lower that level (the init program is described in Section 14.6). Security levels are defined as follows:

-1. Permanently insecure mode: Always run system in level 0 mode (must be compiled into the kernel).

0. Insecure mode: Immutable and append-only flags may be turned off. All devices can be read or written, subject to their permissions.

1. Secure mode: The superuser-settable immutable and append-only flags cannot be cleared; disks for mounted filesystems and kernel memory (/dev/mem and /dev/kmem) are read-only.

2. Highly secure mode: This mode is the same as secure mode, except that disks are always read-only whether mounted or not. This level precludes even a superuser process from tampering with filesystems by unmounting them, but it also inhibits formatting of new filesystems.

Normally, the system runs with level 0 security while in single-user mode, and with level 1 security while in multiuser mode. If level 2 security is desired while the system is running in multiuser mode, it should be set in the /etc/rc startup script (the /etc/rc script is described in Section 14.6).

Files marked immutable by the superuser cannot be changed except by someone with physical access to either the machine or the system console. Files marked immutable include those that are frequently the subject of attack by intruders (e.g., login and su). The append-only flag is typically used for critical system logs. If an intruder breaks in, he will be unable to cover his tracks. Although simple in concept, these two features improve the security of a system dramatically.

One change in the UFS2 inode format was to split the flags field into two separate 32-bit fields: one for flags that can be set by applications (as in UFS1) and a new field for flags maintained strictly by the kernel. An example of a kernel flag is the SNAPSHOT flag used to label a file as being a snapshot. Another kernel-only flag is OPAQUE, which is used by the union filesystem to mark a directory that should not make the layers below it visible. By moving these kernel flags from the high 16 bits of the user-flags field into a separate kernel-flags field, they will not be accidentally set or cleared by a naive or malicious application.

Dynamic Inodes

A common complaint about the UFS1 filesystem is that it preallocates all its inodes at the time that the filesystem is created. For filesystems with millions of files, the initialization of the filesystem can take several hours. Additionally, the filesystem creation program, newfs, had to assume that every filesystem would be filled with many small files and allocate a lot more inodes than were likely to ever be used. If a UFS1 filesystem uses up all its inodes, the only way to get more is to dump, rebuild, and restore the filesystem. The UFS2 filesystem resolves these problems by dynamically allocating its inodes. The usual implementation of dynamically allocated inodes requires a separate filesystem data structure (typically referred to as the inode file) that tracks the current set of inodes. The management and maintenance of this extra data structure adds overhead and complexity and often degrades performance.

To avoid these costs, UFS2 preallocates a range of inode numbers and a set of blocks for each cylinder group (cylinder groups are described in Section 8.9). Initially each cylinder group has two blocks of inodes allocated (a typical block holds 32 or 64 inodes). When the blocks fill up, the next block of inodes in the set is allocated and initialized. The set of blocks that may be allocated to inodes is held as part of the free-space reserve until all other space in the filesystem is allocated. Only then can it be used for file data.

In theory, a filesystem could fill, using up all the blocks set aside for inodes. Later, after large files have been removed and many small files created to replace them, the filesystem might find itself unable to allocated the needed inodes because all the space set aside for inodes was still in use. Here, it would be necessary to reallocate existing files to move them to new locations outside the inode area. Such code has not been written as we do not expect that this condition will arise in practice, since the free space reserve used on most filesystems (8 percent) exceeds the amount of space needed for inodes (typically less than 6 percent). On these systems only a process running with root privileges would ever be able to allocate the inode blocks. Should the code prove necessary in real use, it can be written at that time. Until it is written, filesystems hitting this condition will return an "out of inodes" error on attempts to create new files.

A side benefit of dynamically allocating inodes is that the time to create a new filesystem in UFS2 is about 1 percent of the time that it takes in UFS1. A filesystem that would take one hour to build in a UFS1 format can be built in under a minute in the UFS2 format. While filesystem creations are not a common operation, having them build quickly does matter to the system administrators that have to do such tasks with some regularity.

The cost of dynamically allocating inodes is one extra disk write for every 64 new inodes that are created. Although this cost is low compared to the other costs of creating 64 new files, some systems administrators might want to preallocate more than the minimal number of inodes. If such a demand arises, it would be trivial to add a flag to the newfs program to preallocate additional inodes at the time that the filesystem is created.

Inode Management

Most of the activity in the local filesystem revolves around inodes. As described in Section 6.6, the kernel keeps a list of active and recently accessed vnodes. The decisions regarding how many and which files should be cached are made by the vnode layer based on information about activity across all filesystems. Each local filesystem will have a subset of the system vnodes to manage. Each uses an inode supplemented with some additional information to identify and locate the set of files for which it is responsible. Figure 8.3 shows the location of the inodes within the system.

Figure 8.3. Layout of kernel tables.

[View full size image]

Reviewing the material in Section 6.4, each process has a process open-file table that has slots for up to a system-imposed limit of file descriptors; this table is maintained as part of the process state. When a user process opens a file (or socket), an unused slot is located in the process's open-file table; the small integer file descriptor that is returned on a successful open is an index value into this table.

The per-process file-table entry points to a system open-file entry, which contains information about the underlying file or socket represented by the descriptor. For files, the file table points to the vnode representing the open file. For the local filesystem, the vnode references an inode. It is the inode that identifies the file itself.

The first step in opening a file is to find the file's associated inode. The lookup request is given to the filesystem associated with the directory currently being searched. When the local filesystem finds the name in the directory, it gets the inode number of the associated file. First, the filesystem searches its collection of inodes to see whether the requested inode is already in memory. To avoid doing a linear scan of all its entries, the system keeps a set of hash chains keyed on inode number and filesystem identifier; see Figure 8.4. If the inode is not in the table, such as the first time a file is opened, the filesystem must request a new vnode. When a new vnode is allocated to the local filesystem, a new structure to hold the inode is allocated.

Figure 8.4. Structure of the inode table.

The next step is to locate the disk block containing the inode and to read that block into a buffer in system memory. When the disk I/O completes, the inode is copied from the disk buffer into the newly allocated inode entry. In addition to the information contained in the disk portion of the inode, the inode table itself maintains supplemental information while the inode is in memory. This information includes the hash chains described previously, as well as flags showing the inode's status, reference counts on its use, and information to manage locks. The information also contains pointers to other kernel data structures of frequent interest, such as the superblock for the filesystem containing the inode.

When the last reference to a file is closed, the local filesystem is notified that the file has become inactive. When it is inactivated, the inode times will be updated, and the inode may be written to disk. However, it remains on the hash list so that it can be found if it is reopened. After being inactive for a period determined by the vnode layer based on demand for vnodes in all the filesystems, the vnode will be reclaimed. When a vnode for a local file is reclaimed, the inode is removed from the previous filesystem's hash chain, and, if the inode is dirty, its contents are written back to disk. The space for the inode is then deallocated, so that the vnode will be ready for use by a new filesystem client.


8.3. Naming

Filesystems contain files, most of which contain ordinary data. Certain files are distinguished as directories and contain pointers to files that may themselves be directories. This hierarchy of directories and files is organized into a tree structure; Figure 8.5 (on page 308) shows a small filesystem tree. Each of the circles in the figure represents an inode with its corresponding inode number inside. Each of the arrows represents a name in a directory. For example, inode 4 is the /usr directory with entry ., which points to itself, and entry .., which points to its parent, inode 2, the root of the filesystem. It also contains the name bin, which references directory inode 7, and the name foo, which references file inode 6.

Figure 8.5. A small filesystem tree.

Directories

Directories are allocated in units called chunks; Figure 8.5 shows a typical directory chunk. The size of a chunk is chosen such that each allocation can be transferred to disk in a single operation. The ability to change a directory in a single operation makes directory updates atomic. Chunks are broken up into variable-length directory entries to allow filenames to be of nearly arbitrary length. No directory entry can span multiple chunks. The first four fields of a directory entry are of fixed length and contain the following:

  1. The inode number, an index into a table of on-disk inode structures; the selected entry describes the file (inodes were described in Section 8.2)

  2. The size of the entry in bytes

  3. The type of the entry

  4. The length of the filename contained in the entry in bytes

The remainder of an entry is of variable length and contains a null-terminated filename padded to a 4-byte boundary. The maximum length of a filename in a directory is 255 characters.

The filesystem records free space in a directory by having entries accumulate the free space in their size fields. Thus, some directory entries are larger than required to hold the entry name plus fixed-length fields. Space allocated to a directory should always be accounted for completely by the total of the sizes of the directory's entries. When an entry is deleted from a directory, the system coalesces the entry's space into the previous entry in the same directory chunk by increasing the size of the previous entry by the size of the deleted entry. If the first entry of a directory chunk is free, then the pointer to the entry's inode is set to zero to show that the entry is unallocated.

Figure 8.6. Format of directory chunks.

[View full size image]

When creating a new directory entry, the kernel must scan the entire directory to ensure that the name does not already exist. While doing this scan it also checks each directory block to see if it has enough space in which to put the new entry. The space need not be contiguous. The kernel will compact the valid entries within a directory block to coalesce several small unused spaces together into a single space that is big enough to hold the new entry. The first block that has enough space is used. The kernel will neither compact space across directory blocks nor create an entry that spans two directory blocks as it always wants to be able to do directory updates by writing a single directory block. If no space is found when scanning the directory, a new block is allocated at the end of the directory.

Applications obtain chunks of directories from the kernel by using the getdirentries system call. For the local filesystem, the on-disk format of directories is identical to that expected by the application, so the chunks are returned uninterpreted. When directories are read over the network or from non-BSD filesystems, such as Apple's HFS, the getdirentries system call has to convert the on-disk representation of the directory to that described.

Normally, programs want to read directories one entry at a time. This interface is provided by the directory-access routines. The opendir() function returns a structure pointer that is used by readdir() to get chunks of directories using getdirentries; readdir() returns the next entry from the chunk on each call. The closedir() function deallocates space allocated by opendir() and closes the directory. In addition, there is the rewinddir() function to reset the read position to the beginning, the telldir() function that returns a structure describing the current directory position, and the seekdir() function that returns to a position previously obtained with telldir().

The UFS1 filesystem uses 32-bit inode numbers. While it is tempting to increase these inode numbers to 64 bits in UFS2, doing so would require that the directory format be changed. There is much code that works directly on directory entries. Changing directory formats would entail creating many more filesystemspecific functions that would increase the complexity and maintainability issues with the code. Furthermore, the current APIs for referencing directory entries use 32-bit inode numbers. So even if the underlying filesystem supported 64-bit inode numbers, they could not currently be made visible to user applications. In the short term, applications are not running into the 4 billion-files-per-filesystem limit that 32-bit inode numbers impose. If we assume that the growth rate in the number of files per filesystem over the last 20 years will continue at the same rate, we estimate that the 32-bit inode number should be enough for another 10 to 20 years. However, the limit will be reached before the 64-bit block limit of UFS2 is reached, so the UFS2 filesystem has reserved a flag in the superblock to show that it is a filesystem with 64-bit inode numbers. When the time comes to begin using 64-bit inode numbers, the flag can be turned on and the new directory format can be used. Kernels that predate the introduction of 64-bit inode numbers check this flag and will know that they cannot mount such filesystems.

Finding of Names in Directories

A common request to the filesystem is to lookup a specific name in a directory. The kernel usually does the lookup by starting at the beginning of the directory and going through, comparing each entry in turn. First, the length of the sought-after name is compared with the length of the name being checked. If the lengths are identical, a string comparison of the name being sought and the directory entry is made. If they match, the search is complete; if they fail, either in the length or in the string comparison, the search continues with the next entry. Whenever a name is found, its name and containing directory are entered into the systemwide name cache described in Section 6.6. Whenever a search is unsuccessful, an entry is made in the cache showing that the name does not exist in the particular directory. Before starting a directory scan, the kernel looks for the name in the cache. If either a positive or negative entry is found, the directory scan can be avoided.

Another common operation is to lookup all the entries in a directory. For example, many programs do a stat system call on each name in a directory in the order that the names appear in the directory. To improve performance for these programs, the kernel maintains the directory offset of the last successful lookup for each directory. Each time that a lookup is done in that directory, the search is started from the offset at which the previous name was found (instead of from the beginning of the directory). For programs that step sequentially through a directory with n files, search time decreases from Order(n2) to Order(n).

One quick benchmark that demonstrates the maximum effectiveness of the cache is running the ls -l command on a directory containing 600 files. On a system that retains the most recent directory offset, the amount of system time for this test is reduced by 85 percent. Unfortunately, the maximum effectiveness is much greater than the average effectiveness. Although the cache is 90 percent effective when hit, it is applicable to only about 25 percent of the names being looked up. Despite the amount of time spent in the lookup routine itself decreasing substantially, the improvement is diminished because more time is spent in the routines that that routine calls. Each cache miss causes a directory to be accessed twice—once to search from the middle to the end and once to search from the beginning to the middle.

These caches provide good directory lookup performance but are ineffective for large directories that have a high rate of entry creation and deletion. Each time a new directory entry is created, the kernel must scan the entire directory to ensure that the entry does not already exist. When an existing entry is deleted, the kernel must scan the directory to find the entry to be removed. For directories with many entries these linear scans are time-consuming.

The approach to solving this problem in FreeBSD 5.2 is to introduce dynamic directory hashing that retrofits a directory indexing system to UFS [Dowse & Malone, 2002]. To avoid repeated linear searches of large directories, the dynamic directory hashing builds a hash table of directory entries on the fly when the directory is first accessed. This table avoids directory scans on later lookups, creates, and deletes. Unlike filesystems originally designed with large directories in mind, these indices are not saved on disk and so the system is backward compatible. The drawback is that the indices need to be built the first time that a large directory is encountered after each system reboot. The effect of the dynamic directory hashing is that large directories in UFS cause minimal performance problems.

When we built UFS2, we contemplated solving the large directory update problem by changing to a more complex directory structure such as one that uses B-trees. This technique is used in many modern filesystems such as XFS [Sweeney et al., 1996], JFS [Best & Kleikamp, 2003], ReiserFS [Reiser, 2001], and in later versions of Ext2 [Phillips, 2001]. We decided not to make the change at the time that UFS2 was first implemented for several reasons. First, we had limited time and resources, and we wanted to get something working and stable that could be used in the time frame of FreeBSD 5.2. By keeping the same directory format, we were able to reuse all the directory code from UFS1, did not have to change numerous filesystem utilities to understand and maintain a new directory format, and were able to produce a stable and reliable filesystem in the time frame available to us. The other reason that we felt that we could retain the existing directory structure is because of the dynamic directory hashing that was added to FreeBSD.

Borrowing the technique used by the Ext2 filesystem a flag was also added to show that an on-disk indexing structure is supported for directories [Phillips, 2001]. This flag is unconditionally turned off by the existing implementation of UFS. In the future, if an implementation of an on-disk directory-indexing structure is added, the implementations that support it will not turn the flag off. Index-supporting kernels will maintain the indices and leave the flag on. If an old non-index-supporting kernel is run, it will turn off the flag so that when the filesystem is once again run under a new kernel, the new kernel will discover that the indexing flag has been turned off and will know that the indices may be out date and have to be rebuilt before being used. The only constraint on an implementation of the indices is that they have to be an auxiliary data structure that references the old linear directory format.

Pathname Translation

We are now ready to describe how the filesystem looks up a pathname. The small filesystem introduced in Figure 8.5 is expanded to show its internal structure in Figure 8.7. Each of the files in Figure 8.5 is shown expanded into its constituent inode and data blocks. As an example of how these data structures work, consider how the system finds the file /usr/bin/vi. It must first search the root directory of the filesystem to find the directory usr. It first finds the inode that describes the root directory. By convention, inode 2 is always reserved for the root directory of a filesystem; therefore, the system finds and brings inode 2 into memory. This inode shows where the data blocks are for the root directory. These data blocks must also be brought into memory so that they can be searched for the entry usr. Having found the entry for usr, the system knows that the contents of usr are described by inode 4. Returning once again to the disk, the system fetches inode 4 to find where the data blocks for usr are located. Searching these blocks, it finds the entry for bin. The bin entry points to inode 7. Next, the system brings in inode 7 and its associated data blocks from the disk to search for the entry for vi. Having found that vi is described by inode 9, the system can fetch this inode and the blocks that contain the vi binary.

Figure 8.7. Internal structure of a small filesystem.

Links

As shown in Figure 8.8, each file has a single inode, but multiple directory entries in the same filesystem may reference that inode (i.e., the inode may have multiple names). Each directory entry creates a hard link of a filename to the inode that describes the file's contents. The link concept is fundamental; inodes do not reside in directories but exist separately and are referenced by links. When all the links to an inode are removed, the inode is deallocated. If one link to a file is removed and the filename is recreated with new contents, the other links will continue to point to the old inode. Figure 8.8 shows two different directory entries, foo and bar, that reference the same file; thus, the inode for the file shows a reference count of 2.

Figure 8.8. Hard links to a file.

The system also supports a symbolic link, or soft link. A symbolic link is implemented as a file that contains a pathname. When the system encounters a symbolic link while looking up a component of a pathname, the contents of the symbolic link are prepended to the rest of the pathname; the lookup continues with the resulting pathname. If a symbolic link contains an absolute pathname, that absolute pathname is used. Otherwise, the contents of the symbolic link are evaluated relative to the location of the link in the file hierarchy (not relative to the current working directory of the calling process).

A symbolic link is illustrated in Figure 8.9. Here, there is a hard link, foo, that points to the file. The other reference, bar, points to a different inode whose contents are a pathname of the referenced file. When a process opens bar, the system interprets the contents of the symbolic link as a pathname to find the file the link references. Symbolic links are treated like data files by the system, rather than as part of the filesystem structure; thus, they can point at directories or files on other filesystems. If a filename is removed and replaced, any symbolic links that point to it will access the new file. Finally, if the filename is not replaced, the symbolic link will point at nothing, and any attempt to access it will return an error.

Figure 8.9. Symbolic link to a file.

When open is applied to a symbolic link, it returns a file descriptor for the file pointed to, not for the link itself. Otherwise, it would be necessary to use indirection to access the file pointed to—and that file, rather than the link, is what is usually wanted. For the same reason, most other system calls that take pathname arguments also follow symbolic links. Sometimes, it is useful to be able to detect a symbolic link when traversing a filesystem or when making an archive tape. So the lstat system call is available to get the status of a symbolic link, instead of the object at which that link points.

A symbolic link has several advantages over a hard link. Since a symbolic link is maintained as a pathname, it can refer to a directory or to a file on a different filesystem. So that loops in the filesystem hierarchy are prevented, unprivileged users are not permitted to create hard links (other than . and ..) that refer to a directory. The design of hard links prevents them from referring to files on a different filesystem.

There are several interesting implications of symbolic links. Consider a process that has current working directory /usr/keith and does cd src, where src is a symbolic link to directory /usr/src. If the process then does a cd .., the current working directory for the process will be in /usr instead of in /usr/keith, as it would have been if src was a normal directory instead of a symbolic link. The kernel could be changed to keep track of the symbolic links that a process has traversed and to interpret .. differently if the directory has been reached through a symbolic link. There are two problems with this implementation. First, the kernel would have to maintain a potentially unbounded amount of information. Second, no program could depend on being able to use .., since it could not be sure how the name would be interpreted.

Many shells keep track of symbolic-link traversals. When the user changes directory through .. from a directory that was entered through a symbolic link, the shell returns the user to the directory from which they came. Although the shell might have to maintain an unbounded amount of information, the worst that will happen is that the shell will run out of memory. Having the shell fail will affect only the user silly enough to traverse endlessly through symbolic links. Tracking of symbolic links affects only change-directory commands in the shell; programs can continue to depend on .. to reference its true parent. Thus, tracking symbolic links outside the kernel in a shell is reasonable.

Since symbolic links may cause loops in the filesystem, the kernel prevents looping by allowing at most eight symbolic link traversals in a single pathname translation. If the limit is reached, the kernel produces an error (ELOOP).


8.4. Quotas

Resource sharing always has been a design goal for the BSD system. By default, any single user can allocate all the available space in the filesystem. In certain environments, uncontrolled use of disk space is unacceptable. Consequently, FreeBSD includes a quota mechanism to restrict the amount of filesystem resources that a user or members of a group can obtain. The quota mechanism sets limits on both the number of files and the number of disk blocks that a user or members of a group may allocate. Quotas can be set separately for each user and group on each filesystem.

Quotas support both hard and soft limits. When a process exceeds its soft limit, a warning is printed on the user's terminal; the offending process is not prevented from allocating space unless it exceeds its hard limit. The idea is that users should stay below their soft limit between login sessions but may use more resources while they are active. If a user fails to correct the problem for longer than a grace period, the soft limit starts to be enforced as the hard limit. The grace period is set by the system administrator and is seven days by default. These quotas are derived from a larger resource-limit package that was developed at the University of Melbourne in Australia by Robert Elz [Elz, 1984].

Quotas connect into the system primarily as an adjunct to the allocation routines. When a new block is requested from the allocation routines, the request is first validated by the quota system with the following steps:

1. If there is a user quota associated with the file, the quota system consults the quota associated with the owner of the file. If the owner has reached or exceeded their limit, the request is denied.

2. If there is a group quota associated with the file, the quota system consults the quota associated with the group of the file. If the group has reached or exceeded its limit, the request is denied.

3. If the quota tests pass, the request is permitted and is added to the usage statistics for the file.

When either a user or group quota would be exceeded, the allocator returns a failure as though the filesystem were full. The kernel propagates this error up to the process doing the write system call.

Quotas are assigned to a filesystem after it has been mounted. A system call associates a file containing the quotas with the mounted filesystem. By convention, the file with user quotas is named quota.user, and the file with group quotas is named quota.group. These files typically reside either in the root of the mounted filesystem or in the /var/quotas directory. For each quota to be imposed, the system opens the appropriate quota file and holds a reference to it in the mount-table entry associated with the mounted filesystem. Figure 8.10 shows the mount-table reference. Here, the root filesystem has a quota on users but has none on groups. The /usr filesystem has quotas imposed on both users and groups. As quotas for different users or groups are needed, they are taken from the appropriate quota file.

Figure 8.10. References to quota files.

Quota files are maintained as an array of quota records indexed by user or group identifiers; Figure 8.11 shows a typical record in a user quota file. To find the quota for user identifier i, the system seeks to the offset i x sizeof(quota structure) in the quota file and reads the quota structure at that offset. Each quota structure contains the limits imposed on the user for the associated filesystem. These limits include the hard and soft limits on the number of blocks and inodes that the user may have, the number of blocks and inodes that the user currently has allocated, and the time that the soft limit should start being enforced as the hard limit. The group quota file works in the same way, except that it is indexed by group identifier.

Figure 8.11. Contents of a quota record.

Active quotas are held in system memory in a data structure known as a dquot entry; Figure 8.12 (on page 318) shows two typical entries. In addition to the quota limits and usage extracted from the quota file, the dquot entry maintains information about the quota while the quota is in use. This information includes fields to allow fast access and identification. Quotas are checked by the chkdq() routine. Since quotas may have to be updated on every write to a file, chkdq() must be able to find and manipulate them quickly. Thus, the task of finding the dquot structure associated with a file is done when the file is first opened for writing. When an access check is done to check for writing, the system checks to see whether there is either a user or a group quota associated with the file. If one or more quotas exist, the inode is set up to hold a reference to the appropriate dquot structures for as long as the inode is resident. The chkdq() routine can determine that a file has a quota simply by checking whether the dquot pointer is nonnull; if it is, all the necessary information can be accessed directly. If a user or a group has multiple files open on the same filesystem, all inodes describing those files point to the same dquot entry. Thus, the number of blocks allocated to a particular user or a group can always be known easily and consistently.

Figure 8.12. Dquot entries.

The number of dquot entries in the system can grow large. To avoid doing a linear scan of all the dquot entries, the system keeps a set of hash chains keyed on the filesystem and on the user or group identifier. Even with hundreds of dquot entries, the kernel needs to inspect only about five entries to determine whether a requested dquot entry is memory resident. If the dquot entry is not resident, such as the first time a file is opened for writing, the system must reallocate a dquot entry and read in the quota from disk. The dquot entry is reallocated from the least-recently used dquot entry. So that it can find the oldest dquot entry quickly, the system keeps unused dquot entries linked together in an LRU chain. When the reference count on a dquot structure drops to zero, the system puts that dquot onto the end of the LRU chain. The dquot structure is not removed from its hash chain, so if the structure is needed again soon, it can still be located. Only when a dquot structure is recycled with a new quota record is it removed and relinked into the hash chain. The dquot entry on the front of the LRU chain yields the least-recently used dquot entry. Frequently used dquot entries are reclaimed from the middle of the LRU chain and are relinked at the end after use.

The hashing structure allows dquot structures to be found quickly. However, it does not solve the problem of how to discover that a user has no quota on a particular filesystem. If a user has no quota, a lookup for the quota will fail. The cost of going to disk and reading the quota file to discover that the user has no quota imposed would be prohibitive. To avoid doing this work each time that a new file is accessed for writing, the system maintains nonquota dquot entries. When an inode owned by a user or group that does not already have a dquot entry is first accessed, a dummy dquot entry is created that has infinite values filled in for the quota limits. When the chkdq() routine encounters such an entry, it will update the usage fields but will not impose any limits. When the user later writes other files, the same dquot entry will be found, thus avoiding additional access to the on-disk quota file. Ensuring that a file will always have a dquot entry improves the performance of writing data, since chkdq() can assume that the dquot pointer is always valid, rather than having to check the pointer before every use.

Quotas are written back to the disk when they fall out of the cache, whenever the filesystem does a sync, or when the filesystem is unmounted. If the system crashes, leaving the quotas in an inconsistent state, the system administrator must run the quotacheck program to rebuild the usage information in the quota files.


8.5. File Locking

Locks may be placed on any arbitrary range of bytes within a file. These semantics are supported in FreeBSD by a list of locks, each of which describes a lock of a specified byte range. An example of a file containing several range locks is shown in Figure 8.13 (on page 320). The list of currently held or active locks appears across the top of the figure, headed by the i_lockf field in the inode, and linked together through the lf_next field of the lock structures. Each lock structure identifies the type of the lock (exclusive or shared), the byte range over which the lock applies, and the identity of the lock holder. A lock may be identified either by a pointer to a process entry or by a pointer to a file entry. A process pointer is used for POSIX-style range locks; a file-entry pointer is used for BSD-style whole file locks. The examples in this section show the identity as a pointer to a process entry. In this example, there are three active locks: an exclusive lock held by process 1 on bytes 1 to 3, a shared lock held by process 2 on bytes 7 to 12, and a shared lock held by process 3 on bytes 7 to 14.

Figure 8.13. A set of range locks on a file.

In addition to the active locks, there are other processes that are sleeping waiting to get a lock applied. Pending locks are headed by the lf_block field of the active lock that prevents them from being applied. If there are multiple pending locks, they are linked through their lf_block fields. New lock requests are placed at the end of the list; thus, processes tend to be granted locks in the order that they requested the locks. Each pending lock uses its lf_next field to identify the active lock that currently blocks it. In the example in Figure 8.13, the first active lock has two other locks pending. There is also a pending request for the range 9 to 12 that is currently linked onto the second active entry. It could equally well have been linked onto the third active entry, since the third entry also blocks it. When an active lock is released, all pending entries for that lock are awakened, so they can retry their request. If the second active lock were released, the result would be that its currently pending request would move over to the blocked list for the last active entry.

A problem that must be handled by the locking implementation is the detection of potential deadlocks. To see how deadlock is detected, consider the addition of the lock request by process 2 outlined in the dashed box in Figure 8.13. Since the request is blocked by an active lock, process 2 must sleep, waiting for the active lock on range 1 to 3 to clear. We follow the lf_next pointer from the requesting lock (the one in the dashed box), to identify the active lock for the 1-to-3 range as being held by process 1. The wait channel for process 1 shows that process 2 is sleeping, waiting for a lock to clear, and identifies the pending lock structure as the pending lock (range 9 to 12) hanging off the lf_block field of the second active lock (range 7 to 12). We follow the lf_next field of this pending lock structure (range 9 to 12) to the second active lock (range 7 to 12) that is held by the lock requester, process 2. Thus, the lock request is denied, as it would lead to a deadlock between processes 1 and 2. This algorithm works on cycles of locks and processes of arbitrary size. Performance is reasonable provided there are fewer than 50 processes contending for locks within the same range of a file.

As we note, the pending request for the range 9 to 12 could equally well have been hung off the third active lock for the range 7 to 14. Had it been, the request for adding the lock in the dashed box would have succeeded, since the third active lock is held by process 3 rather than by process 2. If the next lock request on this file were to release the third active lock, then deadlock detection would occur when process 1's pending lock got shifted to the second active lock (range 7 to 12). The difference is that process 1, instead of process 2, would get the deadlock error.

When a new lock request is made, it must first be checked to see whether it is blocked by existing locks held by other processes. If it is not blocked by other processes, it must then be checked to see whether it overlaps any existing locks already held by the process making the request. There are five possible overlap cases that must be considered; these possibilities are shown in Figure 8.14. The assumption in the figure is that the new request is of a type different from that of the existing lock (i.e., an exclusive request against a shared lock, or vice versa). If the existing lock and the request are of the same type, the analysis is a bit simpler. The five cases are as follows:

  1. The new request exactly overlaps the existing lock. The new request replaces the existing lock. If the new request downgrades from exclusive to shared, all requests pending on the old lock are awakened.

  2. The new request is a subset of the existing lock. The existing lock is broken into three pieces (two if the new lock begins at the beginning or ends at the end of the existing lock). If the type of the new request differs from that of the existing lock, all requests pending on the old lock are awakened, so they can be reassigned to the correct new piece, blocked on a lock held by some other process, or granted.

  3. The new request is a superset of an existing lock. The new request replaces the existing lock. If the new request downgrades from exclusive to shared, all requests pending on the old lock are awakened.

  4. The new request extends past the end of an existing lock. The existing lock is shortened, and its overlapped piece is replaced by the new request. All requests pending on the existing lock are awakened, so they can be reassigned to the correct new piece, blocked on a lock held by some other process, or granted.

  5. The new request extends into the beginning of an existing lock. The existing lock is shortened, and its overlapped piece is replaced by the new request. All requests pending on the existing lock are awakened, so they can be reassigned to the correct new piece, blocked on a lock held by some other process, or granted.

Figure 8.14. Five types of overlap considered by the kernel when a range lock is added.

[View full size image]

In addition to the five basic types of overlap outlined, a request may span several existing locks. Specifically, a new request may be composed of zero or one of type 4, zero or more of type 3, and zero or one of type 5.

To understand how the overlap is handled, we can consider the example shown in Figure 8.15. This figure shows a file that has all its active range locks held by process 1, plus a pending lock for process 2.

Figure 8.15. Locks before addition of exclusive-lock request by process 1 on range 3..13.

[View full size image]

Now consider a request by process 1 for an exclusive lock on the range 3 to 13. This request does not conflict with any active locks (because all the active locks are already held by process 1). The request does overlap all three active locks, so the three active locks represent a type 4, type 3, and type 5 overlap, respectively. The result of processing the lock request is shown in Figure 8.16. The first and third active locks are trimmed back to the edge of the new request, and the second lock is replaced entirely. The request that had been held pending on the first lock is awakened. It is no longer blocked by the first lock but is blocked by the newly installed lock. So it now hangs off the blocked list for the second lock. The first and second locks could have been merged because they are of the same type and are held by the same process. However, the current implementation makes no effort to do such merges because range locks are normally released over the same range that they were created. If the merge were done, it would probably have to be split again when the release was requested.

Figure 8.16. Locks after addition of exclusive-lock request by process 1 on range 3.13.

[View full size image]

Lock-removal requests are simpler than addition requests; they need only to consider existing locks held by the requesting process. Figure 8.17 shows the five possible ways that a removal request can overlap the locks of the requesting process.

  1. The unlock request exactly overlaps an existing lock. The existing lock is deleted, and any lock requests that were pending on that lock are awakened.

  2. The unlock request is a subset of an existing lock. The existing lock is broken into two pieces (one if the unlock request begins at the beginning or ends at the end of the existing lock). Any locks that were pending on that lock are awakened so that they can be reassigned to the correct new piece, blocked on a lock held by some other process, or granted.

  3. The unlock request is a superset of an existing lock. The existing lock is deleted, and any locks that were pending on that lock are awakened.

  4. The unlock request extends past the end of an existing lock. The end of the existing lock is shortened. Any locks that were pending on that lock are awakened so that they can be reassigned to the shorter lock, blocked on a lock held by some other process, or granted.

  5. The unlock request extends into the beginning of an existing lock. The beginning of the existing lock is shortened. Any locks that were pending on that lock are awakened so that they can be reassigned to the shorter lock, blocked on a lock held by some other process, or granted.

Figure 8.17. Five types of overlap considered by the kernel when a range lock is deleted.

[View full size image]

In addition to the five basic types of overlap outlined, an unlock request may span several existing locks. Specifically, a new request may be composed of zero or one of type 4, zero or more of type 3, and zero or one of type 5.


8.6. Soft Updates

In filesystems, metadata (e.g., directories, inodes, and free block maps) gives structure to raw storage capacity. Metadata provides pointers and descriptions for linking multiple disk sectors into files and identifying those files. To be useful for persistent storage, a filesystem must maintain the integrity of its metadata in the face of unpredictable system crashes, such as power interruptions and operating system failures. Because such crashes usually result in the loss of all information in volatile main memory, the information in nonvolatile storage (i.e., disk) must always be consistent enough to deterministically reconstruct a coherent filesystem state. Specifically, the on-disk image of the filesystem must have no dangling pointers to uninitialized space, no ambiguous resource ownership caused by multiple pointers, and no unreferenced live resources. Maintaining these invariants generally requires sequencing (or atomic grouping) of updates to small on-disk metadata objects.

Traditionally, the UFS filesystem used synchronous writes to properly sequence stable storage changes. For example, creating a file involves first allocating and initializing a new inode and then filling in a new directory entry to point to it. With the synchronous write approach, the filesystem forces an application that creates a file to wait for the disk write that initializes the on-disk inode. As a result, filesystem operations like file creation and deletion proceed at disk speeds rather than processor/memory speeds [McVoy & Kleiman, 1991; Ousterhout, 1990; Seltzer et al., 1993]. Since disk access times are long compared to the speeds of other computer components, synchronous writes reduce system performance.

The metadata update problem can also be addressed with other mechanisms. For example, one can eliminate the need to keep the on-disk state consistent by using NVRAM technologies, such as an uninterruptible power supply or Flash RAM [Moran et al., 1990; Wu & Zwaenepoel, 1994]. Filesystem operations can proceed as soon as the block to be written is copied into the stable store, and updates can propagate to disk in any order and whenever it is convenient. If the system fails, unfinished disk operations can be completed from the stable store when the system is rebooted.

Another approach is to group each set of dependent updates as an atomic operation with some form of write-ahead logging [Chutani et al., 1992; Hagmann, 1987] or shadow-paging [Chamberlin & Astrahan, 1981; Rosenblum & Ousterhout, 1992; Stonebraker, 1987]. These approaches augment the on-disk state with a log of filesystem updates on a separate disk or in stable store. Filesystem operations can then proceed as soon as the operation to be done is written into the log. If the system fails, unfinished filesystem operations can be completed from the log when the system is rebooted. Many modern filesystems successfully use write-ahead logging to improve performance compared to the synchronous write approach.

In Ganger & Patt [1994], an alternative approach called soft updates was proposed and evaluated in the context of a research prototype. Following a successful evaluation, a production version of soft updates was written for FreeBSD. With soft updates, the filesystem uses delayed writes (i.e., write-back caching) for metadata changes, tracks dependencies between updates, and enforces these dependencies at write-back time. Because most metadata blocks contain many pointers, cyclic dependencies occur frequently when dependencies are recorded only at the block level. Therefore, soft updates track dependencies on a per-pointer basis, which allows blocks to be written in any order. Any still-dependent updates in a metadata block are rolled back before the block is written and rolled forward afterwards. Thus, dependency cycles are eliminated as an issue. With soft updates, applications always see the most current copies of metadata blocks, and the disk always sees copies that are consistent with its other contents.

Update Dependencies in the Filesystem

Several important filesystem operations consist of a series of related modifications to separate metadata structures. To ensure recoverability in the presence of unpredictable failures, the modifications often must be propagated to stable storage in a specific order. For example, when creating a new file, the filesystem allocates an inode, initializes it, and constructs a directory entry that points to it. If the system goes down after the new directory entry has been written to disk but before the initialized inode is written, consistency may be compromised, since the contents of the on-disk inode are unknown. To ensure metadata consistency, the initialized inode must reach stable storage before the new directory entry. We refer to this requirement as an update dependency, because safely writing the directory entry depends on first writing the inode. The ordering constraints map onto three simple rules:

  1. Never point to a structure before it has been initialized (e.g., an inode must be initialized before a directory entry references it).

  2. Never reuse a resource before nullifying all previous pointers to it (e.g., an inode's pointer to a data block must be nullified before that disk block may be reallocated for a new inode).

  3. Never reset the old pointer to a live resource before the new pointer has been set (e.g., when renaming a file, do not remove the old name for an inode until after the new name has been written).

There are eight filesystem activities that require update ordering to ensure postcrash recoverability: file creation, file removal, directory creation, directory removal, file/directory rename, block allocation, indirect block manipulation, and free map management.

The two main resources managed by the filesystem are inodes and data blocks. Two bitmaps are used to maintain allocation information about these resources. For each inode in the filesystem, the inode bitmap has a bit that is set if the inode is in use and cleared if it is free. For each block in the filesystem, the data block bitmap has a bit that is set if the block is free and cleared if it is in use. Each filesystem is broken down into fixed-size pieces called cylinder groups (described more fully in Section 8.9). Each cylinder group has a cylinder group block that contains the bitmaps for the inodes and data blocks residing within that cylinder group. For a large filesystem, this organization allows just those sub-pieces of the filesystem bitmap that are actively being used to be brought into the kernel memory. Each of these active cylinder group blocks is stored in a separate I/O buffer and can be written to disk independently of the other cylinder group blocks.

When a file is created, three metadata structures located in separate blocks are modified. The first is a new inode, which is initialized with its type field set to the new file type, its link count set to one to show that it is live (i.e., referenced by some directory entry), its permission fields set as specified, and all other fields set to default values. The second is the inode bitmap, which is modified to show that the inode has been allocated. The third is a new directory entry, which is filled in with the new name and a pointer to the new inode. To ensure that the bitmaps always reflect all allocated resources, the bitmap must be written to disk before the inode or directory entry. Because the inode is in an unknown state until after it has been initialized on the disk, rule #1 specifies that there is an update dependency requiring that the relevant inode be written before the relevant directory entry. Although not strictly necessary, most BSD fast filesystem implementations also immediately write the directory block before the system call creating the file returns. This second synchronous write ensures that the filename is on stable storage if the application later does an fsync system call. If it were not done, then the fsync call would have to be able to find all the unwritten directory blocks containing a name for the file and write them to disk. A similar update dependency between inode and directory entry exists when adding a second name for the same file (a.k.a. a hard link), since the addition of the second name requires the filesystem to increment the link count in the inode and write the inode to disk before the entry may appear in the directory.

When a file is deleted, a directory block, an inode block, and one or more cylinder group bitmaps are modified. In the directory block, the relevant directory entry is "removed" by reclaiming its space or by nullifying the inode pointer. In the inode block, the relevant inode's type field, link count, and block pointers are zeroed out. The deleted file's blocks and inode are then added to the appropriate free block/inode maps. Rule #2 specifies that there are update dependencies between the directory entry and the inode and between the inode and any modified free map bits. To keep the link count conservatively high (and reduce complexity in practice), the update dependency between a directory entry and inode also exists when removing one of multiple names (hard links) for a file.

Creation and removal of directories is largely as just described for regular files. However, the .. entry is a link from the child directory to the parent, which adds additional update dependencies. Specifically, during creation, the parent's link count must be incremented on disk before the new directory's .. pointer is written. Likewise, during removal, the parent's link count must be decremented after the removed directory's .. pointer is nullified. (Note that this nullification is implicit in deleting the child directory's pointer to the corresponding directory block.)

When a new block is allocated, its bitmap location is updated to reflect that it is in use and the block's contents are initialized with newly written data or zeros. In addition, a pointer to the new block is added to an inode or indirect block (see following). To ensure that the on-disk bitmap always reflects allocated resources, the bitmap must be written to disk before the pointer. Also, because the contents of the newly allocated disk location are unknown, rule #1 specifies an update dependency between the new block and the pointer to it. Because enforcing this update dependency with synchronous writes can reduce data creation throughput by a factor of two [Ganger & Patt, 1994], many implementations ignore it for regular data blocks. This implementation decision reduces integrity and security, since newly allocated blocks generally contain previously deleted file data. Soft updates allow all block allocations to be protected in this way with near-zero performance reduction.

Manipulation of indirect blocks does not introduce fundamentally different update dependencies, but they do merit separate discussion. Allocation, both of indirect blocks and of blocks pointed to by indirect blocks, is as just discussed. File deletion, and specifically deallocation, is more interesting for indirect blocks. Because the inode reference is the only way to identify indirect blocks and blocks connected to them (directly or indirectly), nullifying the inode's pointer to an indirect block is enough to eliminate all recoverable pointers to said blocks. Once the pointer is nullified on disk, all its blocks can be freed. The exception to this rule is when a file is partially truncated. Here, the pointer from the inode to the indirect block remains. Some of the indirect block pointers will be zeroed and and their corresponding blocks freed while the rest of the pointers are left intact.

When a file is being renamed, two directory entries are affected. A new entry (with the new name) is created and set to point to the relevant inode, and the old entry is removed. Rule #3 states that the new entry should be written to disk before the old entry is removed to avoid having the file unreferenced on reboot. If link counts are being kept conservatively, rename involves at least four disk updates in sequence: one to increment the inode's link count, one to add the new directory entry, one to remove the old directory entry, and one to decrement the link count. If the new name already existed, then the addition of the new directory entry also acts as the first step of file removal as discussed above. Interestingly, rename is the one POSIX file operation that should have an atomic update to multiple user-visible metadata structures to provide ideal semantics. POSIX does not require said semantics and most implementations, including FreeBSD, cannot provide it.

On an active filesystem, the bitmaps change constantly. Thus, the copy of the bitmaps in the kernel memory often differs from the copy that is stored on the disk. If the system halts without writing out the incore state of the bitmaps, some of the recently allocated inodes and data blocks may not be reflected in the out-of-date copies of the bitmaps on the disk. So the filesystem check program, fsck, must be run over all the inodes in the filesystem to ascertain which inodes and blocks are in use and bring the bitmaps up to date [McKusick & Kowalski, 1994]. An added benefit of soft updates is that it tracks the writing of the bitmaps to disk and uses this information to ensure that no newly allocated inodes or pointers to newly allocated data blocks will be written to disk until after the bitmap that references them has been written to disk. This guarantee ensures that there will never be an allocated inode or data block that is not marked in the on-disk bitmap. This guarantee, together with the other guarantees made by the soft update code, means that it is no longer necessary to run fsck after a system crash.

The next 12 subsections describe the soft updates data structures and their use in enforcing the update dependencies just described. The structures and algorithms described eliminate all synchronous write operations from the filesystem except for the partial truncation of a file and the fsync system call, which explicitly requires that all the state of a file be committed to disk before the system call returns.

The key attribute of soft updates is dependency tracking at the level of individual changes within cached blocks. Thus, for a block containing 64 inodes, the system can maintain up to 64 dependency structures with one for each inode in the buffer. Similarly for a buffer containing a directory block containing 50 names, the system can maintain up to 50 dependency structures with one for each name in the directory. With this level of detailed dependency information, circular dependencies between blocks are not problematic. For example, when the system wishes to write a buffer containing inodes, those inodes that can be safely written can go to the disk. Any inodes that cannot yet be safely written are temporarily rolled back to their safe values while the disk write proceeds. After the disk write completes, such inodes are rolled forward to their current values. Because the buffer is locked throughout the time that the contents are rolled back, the disk write is being done, and the contents are rolled forward, any processes wishing to use the buffer will be blocked from accessing it until it has been returned to its correct state.

Dependency Structures

The soft updates implementation uses a variety of data structures to track pending update dependencies among filesystem structures. Table 8.2 (on page 330) lists the dependency structures used in the BSD soft updates implementation, their main functions, and the types of blocks with which they can be associated. These dependency structures are allocated and associated with blocks as various file operations are completed. They are connected to the in-core blocks with which they are associated by a pointer in the corresponding buffer header. Two common aspects of all listed dependency structures are the worklist structure and the states used to track the progress of a dependency.

Table 8.2. Soft updates and dependency tracking.

Name

Function

Associated Structures

bmsafemap

track bitmap dependencies (points to lists of dependency structures for recently allocated blocks and inodes)

cylinder group block

inodedep

track inode dependencies (information and list head pointers for all inode-related dependencies, including changes to the link count, the block pointers, and the file size)

inode block

allocdirect

track inode-referenced blocks (linked into lists pointed to by an inodedep and a bmsafemap to track inode's dependence on the block and bitmap being written to disk)

data block or indirect block or directory block

indirdep

track indirect block dependencies (points to list of dependency structures for recently allocated blocks with pointers in the indirect block)

indirect block

allocindir

track indirect-block referenced block (linked into lists pointed to by an indirdep and a bmsafemap to track the indirect block's dependence on that block and bitmap being written to disk)

data block or indirect block or directory block

pagedep

track directory block dependencies (points to lists of diradd and dirrem structures)

directory block

diradd

track dependency between a new directory entry and the referenced inode

inodedep and directory block

mkdir

track new directory creation (used in addition to standard diradd structure when doing a mkdir)

inodedep and directory block

dirrem

track dependency between a deleted directory entry and the unlinked inode

first pagedep then tasklist

freefrag

tracks a single block or fragment to be freed as soon as the corresponding block (containing the inode with the now-replaced pointer to it) is written to disk

first inodedep then tasklist

freeblks

tracks all the block pointers to be freed as soon as the corresponding block (containing the inode with the now-zeroed pointers to them) is written to disk

first inodedep then tasklist

freefile

tracks the inode that should be freed as soon as the corresponding block (containing the inode block with the now-reset inode) is written to disk

first inodedep then tasklist

The worklist structure is really just a common header included as the first item in each dependency structure. It contains a set of linkage pointers and a type field to show the type of structure in which it is embedded. The worklist structure allows several different types of dependency structures to be linked together into a single list. The soft updates code can traverse one of these heterogenous lists, using the type field to determine which kind of dependency structure it has encountered, and take the appropriate action with each.

The typical use for the worklist structure is to link together a set of dependencies associated with a buffer. Each buffer in the system has a pointer to a worklist added to it. Any dependencies associated with that buffer are linked onto its worklist. After the buffer has been locked and just before the buffer is to be written, the I/O system passes the buffer to the soft updates code to let it know that a disk write is about to be started. The soft updates code then traverses the list of dependencies associated with the buffer and does any needed roll-back operations. After the disk write completes but before the buffer is unlocked, the I/O system calls the soft updates code to let it know that a write has completed. The soft updates code then traverses the list of dependencies associated with the buffer, does any needed roll-forward operations, and deallocates any dependencies that are fulfilled by the data in the buffer having been written to disk.

Another important list maintained by the soft updates code is the tasklist that contains background tasks for the work daemon. Dependency structures are generally added to the tasklist during the disk write completion routine, describing tasks that have become safe given the disk update but that may need to block for locks or I/O and therefore cannot be completed during the interrupt handler. Once per second, the syncer daemon (in its dual role as the soft updates work daemon) wakes up and calls into the soft updates code to process any items on the tasklist.

The work done for a dependency structure on this list is type-dependent. For example, for a freeblks structure, the listed blocks are marked free in the block bitmaps. For a dirrem structure, the associated inode's link count is decremented, possibly triggering file deletion.

Most dependency structures have a set of flags that describe the state of completion of the corresponding dependency. Dirty cache blocks can be written to the disk at any time. When the I/O system hands the buffer to the soft updates code (before and after a disk write), the states of the associated dependency structures determine what actions are taken. Although the specific meanings vary from structure to structure, the three main flags and their general meanings are:

ATTACHED | The ATTACHED flag shows that the buffer with which the dependency structure is associated is not currently being written. When a disk write is started for a buffer with a dependency that must be rolled back, the ATTACHED flag is cleared in the dependency structure to show that it has been rolled back in the buffer. When the disk write completes, updates described by dependency structures that have the ATTACHED flag cleared are rolled forward, and the ATTACHED flag is set. Thus, a dependency structure can never be deleted while its ATTACHED flag is cleared, since the information needed to do the roll-forward operation would then be lost.
DEPCOMPLETE | The DEPCOMPLETE flag shows that all associated dependencies have been completed. When a disk write is started, the update described by a dependency structure is rolled back if the DEPCOMPLETE flag is clear. For example, in a dependency structure that is associated with newly allocated inodes or data blocks, the DEPCOMPLETE flag is set when the corresponding bitmap has been written to disk.
COMPLETE | The COMPLETE flag shows that the update being tracked has been committed to the disk. For some dependencies, updates will be rolled back during disk writes when the COMPLETE flag is clear. For example, for a newly allocated data block, the COMPLETE flag is set when the contents of the block have been written to disk.

In general, the flags are set as disk writes complete, and a dependency structure can be deallocated only when its ATTACHED, DEPCOMPLETE, and COMPLETE flags are all set. Consider the example of a newly allocated data block that will be tracked by an allocdirect structure. The ATTACHED flag will initially be set when the allocation occurs. The DEPCOMPLETE flag will be set after the bitmap allocating that new block is written. The COMPLETE flag will be set after the contents of the new block are written. If the inode claiming the newly allocated block is written before both the DEPCOMPLETE and COMPLETE flags are set, the ATTACHED flag will be cleared while the block pointer in the inode is rolled back to zero, the inode is written, and the block pointer in the inode is rolled forward to the new block number. Where different, the specific meanings of these flags in the various dependency structures are described in the subsections that follow.

Bitmap Dependency Tracking

Bitmap updates are tracked by the bmsafemap structure shown in Figure 8.18. Each buffer containing a cylinder group block will have its own bmsafemap structure. As with every dependency structure, the first entry in the bmsafemap structure is a worklist structure. Each time an inode, direct block, or indirect block is allocated from the cylinder group, a dependency structure is created for that resource and linked onto the appropriate bmsafemap list. Each newly allocated inode will be represented by an inodedep structure linked to the bmsafemap inodedep head list. Each newly allocated block directly referenced by an inode will be represented by an allocdirect structure linked to the bmsafemap allocdirect head list. Each newly allocated block referenced by an indirect block will be represented by an allocindir structure linked to the bmsafemap allocindir head list. Because of the code's organization, there is a small window between the time a block is first allocated and the time at which its use is known. During this period of time, it is described by a newblk structure linked to the bmsafemap new blk head list. After the kernel chooses to write the cylinder group block, the soft updates code will be notified when the write has completed. At that time, the code traverses the inode, direct block, indirect block, and new block lists, setting the DEPCOMPLETE flag in each dependency structure and removing said dependency structure from its dependency list. Having cleared all its dependency lists, the bmsafemap structure can be deallocated. There are multiple lists as it is slightly faster and more type-safe to have lists of specific types.

Figure 8.18. Bitmap update dependencies.

Inode Dependency Tracking

Inode updates are tracked by the inodedep structure shown in Figure 8.19. The worklist and state fields are as described for dependency structures in general. The filesystem ptr and inode number fields identify the inode in question. When an inode is newly allocated, its inodedep is attached to the inodedep head list of a bmsafemap structure. Here, deps list chains additional new inodedep structures and dep bp points to the cylinder group block that contains the corresponding bitmap. Other inodedep fields are explained in later subsections.

Figure 8.19. Inode update dependencies.

Before detailing the rest of the dependencies associated with an inode, we need to discuss the steps involved in updating an inode on disk as pictured in Figure 8.20.

  1. The kernel calls the vnode operation, VOP_UPDATE, which requests that the disk resident part of an inode (referred to as a dinode) be copied from its in-memory vnode structure to the appropriate disk buffer. This disk buffer holds the contents of an entire disk block, which is usually big enough to include 64 dinodes. Some dependencies are fulfilled only when the inode has been written to disk. These dependencies need dependency structures to track the progress of the writing of the inode. Therefore, during step 1, a soft update routine, softdep_update_inodeblock(), is called to move allocdirect structures from the incore update list to the buffer update list and to move freefile, freeblks, freefrag, diradd, and mkdir structures (described below) from the inode wait list to the buffer wait list.

  2. The kernel calls the vnode operation, VOP_STRATEGY, that prepares to write the buffer containing the dinode, pointed to by bp in Figure 8.20. A soft updates routine, softdep_disk_io_initiation(), identifies inodedep dependencies and calls initiate_write_inodeblock() to do roll-backs as necessary.

  3. Output completes on the buffer referred to by bp and the I/O system calls a routine, biodone(), to notify any waiting processes that the write has finished. The biodone() routine then calls a soft updates routine, softdep_disk_write_complete(), that identifies inodedep dependencies and calls handle_written_inodeblock() to revert roll-backs and clear any dependencies on the buffer wait and buffer update lists.

Figure 8.20. Inode update steps.

Direct Block Dependency Tracking

Figure 8.21 illustrates the dependency structures involved in allocation of direct blocks. Recall that the key dependencies are that, before the on-disk inode points to a newly allocated block, both the corresponding bitmap block and the new block itself must be written to the disk. The order in which the two dependencies complete is not important. The figure introduces the allocdirect structure that tracks blocks directly referenced by the inode. The three recently allocated logical blocks (1, 2, and 3) shown are each in a different state. For logical block 1, the bitmap block dependency is complete (as shown by the DEPCOMPLETE flag being set), but the block itself has not yet been written (as shown by the COMPLETE flag being cleared). For logical block 2, both dependencies are complete. For logical block 3, neither dependency is complete, so the corresponding allocdirect structure is attached to a bmsafemap allocdirect head list (recall that this list is traversed to set DEPCOMPLETE flags after bitmap blocks are written). The COMPLETE flag for logical blocks 1 and 3 will be set when their initialized data blocks are written to disk. The figure also shows that logical block 1 existed at a time that VOP_UPDATE was called, which is why its allocdirect structure resides on the inodedep buffer update list. Logical blocks 2 and 3 were created after the most recent call to VOP_UPDATE and thus their structures reside on the inodedep incore update list.

Figure 8.21. Direct block allocation dependencies.

[View full size image]

For files that grow in small steps, a direct block pointer may first point to a fragment that is later promoted to a larger fragment and eventually to a full-sized block. When a fragment is replaced by a larger fragment or a full-sized block, it must be released back to the filesystem. However, it cannot be released until the new fragment or block has had its bitmap entry and contents written and the inode claiming the new fragment or block has been written to the disk. The fragment to be released is described by a freefrag structure (not shown). The freefrag structure is held on the freefrag list of the allocdirect for the block that will replace it until the new block has had its bitmap entry and contents written. The freefrag structure is then moved to the inode wait list of the inodedep associated with its allocdirect structure where it migrates to the buffer wait list when VOP_UPDATE is called. The freefrag structure eventually is added to the tasklist after the buffer holding the inode block has been written to disk. When the tasklist is serviced, the fragment listed in the freefrag structure is returned to the free-block bitmap.

Indirect Block Dependency Tracking

Figure 8.22 shows the dependency structures involved in allocation of indirect blocks that includes the same dependencies as with direct blocks. This figure introduces two new dependency structures. A separate allocindir structure tracks each individual block pointer in an indirect block. The indirdep structure manages all the allocindir dependencies associated with an indirect block. The figure shows a file that recently allocated logical blocks 14 and 15 (the third and fourth entries, at offsets 8 and 12, in the first indirect block). The allocation bitmaps have been written for logical block 14 (as shown by its DEPCOMPLETE flag being set), but not for block 15. Thus, the bmsafemap structure tracks the allocindir structure for logical block 15. The contents of logical block 15 have been written to disk (as shown by its COMPLETE flag being set), but not those of block 14. The COMPLETE flag will be set in 14's allocindir structure once the block is written.

Figure 8.22. Indirect block allocation dependencies.

[View full size image]

The list of allocindir structures tracked by an indirdep structure can be long (e.g., up to 2048 entries for 8-Kbyte indirect blocks). To avoid traversing lengthy dependency structure lists in the I/O routines, an indirdep structure maintains a second version of the indirect block: the saved data ptr always points to the buffer's up-to-date copy and the safe copy ptr points to a version that includes only the subset of pointers that can be safely written to disk (and NULL for the others). The former is used for all filesystem operations and the latter is used for disk writes. When the allocindir head list becomes empty, the saved data ptr and safe copy ptr point to identical blocks and the indirdep structure (and the safe copy) can be deallocated.

Dependency Tracking for New Indirect Blocks

Figure 8.23 shows the structures associated with a file that recently expanded into its single-level indirect block. Specifically, this expansion involves inodedep and indirdep structures to manage dependency structures for the inode and indirect block, an allocdirect structure to track the dependencies associated with the indirect block's allocation, and an allocindir structure to track the dependencies associated with a newly allocated block pointed to by the indirect block. These structures are used as described in the previous three subsections. Neither the indirect block nor the data block that it references have had their bitmaps set, so they do not have their DEPCOMPLETE flag set and are tracked by a bmsafemap structure. The bitmap entry for the inode has been written, so the inodedep structure has its DEPCOMPLETE flag set. The use of the buffer update head list by the inodedep structure shows that the in-core inode has been copied into its buffer by a call to VOP_UPDATE. Neither of the dependent pointers (from the inode to the indirect block and from the indirect block to the data block) can be safely included in disk writes yet, since the corresponding COMPLETE and DEPCOMPLETE flags are not set. Only after the bitmaps and the contents have been written will all the flags be set and the dependencies complete.

Figure 8.23. Dependencies for a file expanding into an indirect block.

[View full size image]

New Directory Entry Dependency Tracking

Figure 8.24 shows the dependency structures for a directory that has two new entries, foo and bar. This figure introduces two new dependency structures. A separate diradd structure tracks each individual directory entry in a directory block. The pagedep structure manages all the diradd dependencies associated with a directory block. For each new file, there is an inodedep structure and a diradd structure. Both files' inodes have had their bitmap's written to disk, as shown by the DEPCOMPLETE flags being set in their inodedep structures. The inode for foo has been updated with VOP_UPDATE but has not yet been written to disk, as shown by the COMPLETE flag on its inodedep structure not being set and by its diradd structure still being linked onto its buffer wait list. Until the inode is written to disk with its increased link count, the directory entry may not appear on disk. If the directory page is written, the soft updates code will roll back the creation of the new directory entry for foo by setting its inode number to zero. After the disk write completes, the roll-back is reversed by restoring the correct inode number for foo.

Figure 8.24. Dependencies associated with adding new directory entries.

[View full size image]

The inode for bar has been written to disk, as shown by the COMPLETE flag being set in its inodedep and diradd structures. When the inode write completed, the diradd structure for bar was moved from the inodedep buffer wait list to the inodedep pending ops list. The diradd also moved from the pagedep diradd list to the pagedep pending ops list. Since the inode has been written, it is safe to allow the directory entry to be written to disk. The diradd entries remain on the inodedep and pagedep pending ops list until the new directory entry is written to disk. When the entry is written, the diradd structure is freed. One reason to maintain the pending ops list is so that when an fsync system call is done on a file, the kernel is able to ensure that both the file's contents and directory reference(s) are written to disk. The kernel ensures that the reference(s) are written by doing a lookup to see if there is an inodedep for the inode that is the target of the fsync. If it finds an inodedep, it checks to see if it has any diradd dependencies on either its pending ops or buffer wait lists. If it finds any diradd structures, it follows the pointers to their associated pagedep structures and flushes out the directory inode associated with that pagedep. This backtracking recurses on the directory inodedep.

New Directory Dependency Tracking

Figure 8.25 (on page 340) shows the two additional dependency structures involved with creating a new directory. For a regular file, the directory entry can be committed as soon as the newly referenced inode has been written to disk with its increased link count. When a new directory is created, there are two additional dependencies: writing the directory data block containing the . and .. entries (MKDIR_BODY) and writing the parent inode with the increased link count for .. (MKDIR_PARENT). These additional dependencies are tracked by two mkdir structures linked to the associated diradd structure. The soft-updates design dictates that any given dependency will correspond to a single buffer at any given point in time. Thus, two structures are used to track the action of the two different buffers. When each completes, it clears its associated flag in the diradd structure. The MKDIR_PARENT is linked to the inodedep structure for the parent directory. When that directory inode is written, the link count will be updated on disk. The MKDIR_BODY is linked to the buffer that contains the initial contents of the new directory. When that buffer is written, the entries for . and .. will be on disk. The last mkdir to complete sets the DEPCOMPLETE flag in the diradd structure so that the diradd structure knows that these extra dependencies have been completed. Once these extra dependencies have been completed, the handling of the directory diradd proceeds exactly as it would for a regular file.

Figure 8.25. Dependencies associated with adding a new directory.

[View full size image]

All mkdir structures in the system are linked together on a list. This list is needed so that a diradd can find its associated mkdir structures and deallocate them if it is prematurely freed (e.g., if a mkdir system call is immediately followed by a rmdir system call of the same directory). Here, the deallocation of a diradd structure must traverse the list to find the associated mkdir structures that reference it. The deletion would be faster if the diradd structure were simply augmented to have two pointers that referenced the associated mkdir structures. However, these extra pointers would double the size of the diradd structure to speed an infrequent operation.

Directory Entry Removal Dependency Tracking

Figure 8.26 shows the dependency structures involved with the removal of a directory entry. This figure introduces one new dependency structure, the dirrem structure, and a new use for the pagedep structure. A separate dirrem structure tracks each individual directory entry to be removed in a directory block. In addition to previously described uses, pagedep structures associated with a directory block manage all dirrem structures associated with the block. After the directory block is written to disk, the dirrem request is added to the work daemon's tasklist list. For file deletions, the work daemon will decrement the inode's link count by one. For directory deletions, the work daemon will decrement the inode's link count by two, truncate its size to zero, and decrement the parent directory's link count by one. If the inode's link count drops to zero, the resource reclamation activities described in the "file and directory inode reclamation" section are started.

Figure 8.26. Dependencies associated with removing a directory entry.

File Truncation

When a file is truncated to zero length without soft updates enabled, the block pointers in its inode are saved in a temporary list, the pointers in the inode are zeroed, and the inode is synchronously written to disk. When the inode write completes, the list of its formerly claimed blocks are added to the free-block bitmap. With soft updates, the block pointers in the inode being truncated are copied into a freeblks structure, the pointers in the inode are zeroed, and the inode is marked dirty. The freeblks structure is added to the inode wait list, and it migrates to the buffer wait list when VOP_UPDATE is called. The freeblks structure is eventually added to the tasklist after the buffer holding the inode block has been written to disk. When the tasklist is serviced, the blocks listed in the freeblks structure are returned to the free-block bitmap.

File and Directory Inode Reclamation

When the link count on a file or directory drops to zero, its inode is zeroed to show that it is no longer in use. When running without soft updates, the zeroed inode is synchronously written to disk, and the inode is marked as free in the bitmap. With soft updates, information about the inode to be freed is saved in a freefile structure. The freefile structure is added to the inode wait list, and it migrates to the buffer wait list when VOP_UPDATE is called. The freefile structure eventually is added to the tasklist after the buffer holding the inode block has been written to disk. When the tasklist is serviced, the inode listed in the freefile structure is returned to the free inode map.

Directory Entry Renaming Dependency Tracking

Figure 8.27 shows the structures involved in renaming a file. The dependencies follow the same series of steps as those for adding a new file entry, with two variations. First, when a roll-back of an entry is needed because its inode has not yet been written to disk, the entry must be set back to the previous inode number rather than to zero. The previous inode number is stored in a dirrem structure. The DIRCHG flag is set in the diradd structure so that the roll-back code knows to use the old inode number stored in the dirrem structure. The second variation is that, after the modified directory entry is written to disk, the dirrem structure is added to the work daemon's tasklist list so that the link count of the old inode will be decremented as described in the section on "Directory Entry Removal Dependency Tracking."

Figure 8.27. Dependencies associated with renaming a directory entry.

Tracking File Removal Dependencies

This section will go through a short example describing the dependencies that track the removal of a file. These three ordering constraints must be maintained:

  1. The filename in the on-disk copy of the directory must be deleted.

  2. The inode describing the file must be deallocated by zeroing out its on-disk dinode.

  3. The blocks formerly referenced by the inode for the file must be released to the free-space bitmap, and the inode needs to be released to the free-inode bitmap.

These constraints are maintained by soft updates as follows:

  1. The block of the directory containing the name to be deleted is read into a kernel buffer. The entry is deleted by changing the entry that precedes it to point to the entry that follows it (see Section 8.3). Before releasing the buffer, a set of dependencies needs to be constructed, as shown in Figure 8.26. If this deletion is the first dependency for the directory block, it needs to have a pagedep structure allocated that is linked onto the dependency list for the buffer. Next a dirrem structure is allocated that records the inode number of the entry being deleted. The dirrem structure is linked onto the dirrem list of the pagedep structure for the directory block. The buffer is then marked dirty, unlocked, and released.

  2. Some time in the next 30 seconds, the kernel will decide to write the dirty directory buffer. When the write completes, the pagedep associated with the buffer is passed to soft updates for processing. One processing step is to handle each of the dirrem entries. Each dirrem entry causes the inode formerly referenced by the directory to have its reference count decremented by one. If the reference count drops to zero (meaning that the last name for the file was removed), then the inode needs to be deallocated and freed. Before zeroing out the contents of the on-disk dinode, its list of allocated blocks must be saved in a freeblks structure and information needed to free the inode must be saved in a freefile structure. The block of the filesystem containing the dinode to be freed is read into a kernel buffer, as shown in Figure 8.20. The part of the buffer containing the dinode is zeroed out. If the deallocation is the first dependency for the inode, it needs to have an inodedep structure allocated that is linked onto the dependency list for the buffer. The freeblks and freefile structures are linked onto the buffer wait list of the inodedep structure. The buffer is then marked dirty, unlocked, and released. The dirrem structure is freed as is the pagedep structure if it is no longer tracking any dependencies.

  3. Some time in the next 30 seconds, the kernel will decide to write the buffer containing the zeroed dinode. When the write completes, the inodedep associated with the buffer is passed to soft updates for processing. One processing step is to handle each of the buffer wait entries. The handling of the freeblks entry causes all its listed blocks to be marked free in the appropriate cylinder-group bitmaps. The handling of the freefile entry causes the deleted inode to be marked free in the appropriate cylinder-group bitmap. The freeblks and freefile structures are freed as is the inodedep structure if it is no longer tracking any dependencies.

The file has now been completely removed and ceases to be tracked by soft updates.

Fsync Requirements for Soft Updates

The fsync system call requests that the specified file be written to stable storage and that the system call not return until all its associated writes have completed. The task of completing an fsync requires more than simply writing all the file's dirty data blocks to disk. It also requires that any unwritten directory entries that reference the file also be written, as well as any unwritten directories between the file and the root of the filesystem. Simply getting the data blocks to disk can be a major task. First, the system must check to see if the bitmap for the inode has been written, finding the bitmap and writing it if necessary. It must then check for, find, and write the bitmaps for any new blocks in the file. Next, any unwritten data blocks must go to disk. Following the data blocks, any first-level indirect blocks that have newly allocated blocks in them are written, followed by any double indirect blocks, then triple-level indirect blocks. Finally, the inode can be written that will ensure that the contents of the file are on stable store. Ensuring that all names for the file are also on stable store requires data structures that can determine whether there are any uncommitted names and, if so, in which directories they occur. For each directory containing an uncommitted name, the soft updates code must go through the same set of flush operations that it has just done on the file itself.

The addition of extended attribute data to the inode required that the soft updates code be extended so that it could ensure the integrity of these new data blocks. As with the file data blocks, soft updates ensure that the extended data blocks and the bitmaps, which show that they are in use, are written to disk before they are claimed by the inode. Soft updates also ensure that any updated extended attribute data are committed to disk as part of an fsync of the file.

Although the fsync system call must ultimately be done synchronously, this requirement does not mean that the flushing operations must each be done synchronously. Instead, whole sets of bitmaps or data blocks are pushed into the disk queue, and the soft updates code then waits for all the writes to complete. This approach is more efficient because it allows the disk subsystem to sort all the write requests into the most efficient order for writing. Still, the fsync part of the soft updates code generates most of the remaining synchronous writes in the filesystem.

Another issue related to fsync is unmounting of filesystems. Doing an unmount requires finding and flushing all the dirty files that are associated with the filesystem. Flushing the files may lead to the generation of background activity, such as removing files whose reference count drops to zero as a result of their nullified directory entries being written. Thus, the system must be able to find all background activity requests and process them. Even on a quiescent filesystem, several iterations of file flushes followed by background activity may be required. FreeBSD allows for the forcible unmount of a filesystem, which may take place while the filesystem is actively in use. The ability to cleanly suspend operations on an active filesystem is described in Section 8.7.

File Removal Requirements for Soft Updates

For correct operation, a directory's .. entry should not be removed until after the directory is persistently unlinked. Correcting this dependency ordering in the soft updates code introduced a delay of up to two minutes between the time a directory is unlinked and the time that it is really deallocated (when the .. entry is removed). Until the directory's .. entry is really removed, the link count on its parent will not be decremented. Thus, when a user removes one or more directories, the link count of their former parent still reflects their being present for several minutes. This delayed link count decrement not only causes some questions from users, but also causes some applications to break. For example, the rmdir system call will not remove a directory that has a link count over two. This restriction means that a directory that recently had directories removed from it cannot be removed until its former directories have been fully deleted.

To fix these link-count problems, the soft updates implementation augments the inode nlink field with a new field called effnlink. The nlink field is still stored as part of the on-disk metadata and reflects the true link count of the inode. The effnlink field is maintained only in kernel memory and reflects the final value that the nlink field will reach once all its outstanding operations are completed. All interactions with user applications report the value of the effnlink field, which results in the illusion that everything has happened immediately.

When a file is removed on a filesystem running with soft updates, the removal appears to happen quickly, but the process of removing the file and returning its blocks to the free list may take up to several minutes. Before UFS2, the space held by the file did not show up in the filesystem statistics until the removal of the file had been completed. Thus, applications that clean up disk space such as the news expiration program would often vastly overshoot their goal. They work by removing files and then checking to see if enough free space has showed up. Because of the time lag in having the free space recorded, they would remove far too many files. To resolve problems of this sort, the soft updates code now maintains a counter that keeps track of the amount of space that is held by the files that it is in the process of removing. This counter of pending space is added to the actual amount of free space as reported by the kernel (and thus by utilities like df). The result of this change is that free space appears immediately after the unlink system call returns or the rm utility finishes.

The second and related change to soft updates has to do with avoiding false out-of-space errors. When running with soft updates on a nearly full filesystem with high turnover rate (for example, when installing a whole new set of binaries on a root partition), the filesystem can return a filesystem full error even though it reports that it has plenty of free space. The filesystem full message happens because soft updates have not managed to free the space from the old binaries in time for it to be available for the new binaries.

The initial attempt to correct this problem was to simply have the process that wished to allocate space wait for the free space to show up. The problem with this approach is that it often had to wait for up to a minute. In addition to making the application seem intolerably slow, it usually held a locked vnode that could cause other applications to get blocked waiting for it to become available (often referred to as a lock race to the root of the filesystem). Although the condition would clear in a minute or two, users often assumed that their system had hung and would reboot.

To remedy this problem, the solution devised for UFS2 is to co-opt the process that would otherwise be blocked and put it to work helping soft updates process the files to be freed. The more processes trying to allocate space, the more help that is available to soft updates and the faster free blocks begin to appear. Usually in under one second enough space shows up that the processes can return to their original task and complete. The effect of this change is that soft updates can now be used on small nearly full filesystems with high turnover.

Although the common case for deallocation is for all data in a file to be deleted, the truncate system call allows applications to delete only part of a file. This semantic creates slightly more complicated update dependencies, including the need to have deallocation dependencies for indirect blocks and the need to consider partially deleted data blocks. Because it is so uncommon, the soft-updates implementation does not optimize this case; the conventional synchronous write approach is used instead.

One concern with soft updates is the amount of memory consumed by the dependency structures. In daily operation, we have found that the additional dynamic memory load placed on the kernel memory allocation area is about equal to the amount of memory used by vnodes plus inodes. For each 1000 vnodes in the system, the additional peak memory load from soft updates is about 300 Kbytes. The one exception to this guideline occurs when large directory trees are removed. Here, the filesystem code can get arbitrarily far ahead of the on-disk state, causing the amount of memory dedicated to dependency structures to grow without bound. The soft-update code was modified to monitor the memory load for this case and not allow it to grow past a tunable upper bound. When the bound is reached, new dependency structures can only be created at the rate at which old ones are retired. The effect of this limit is to slow down the rate of removal to the rate at which the disk updates can be done. While this restriction slows the rate at which soft updates can normally remove files, it is still considerably faster than the traditional synchronous-write filesystem. In steady-state, the soft-update remove algorithm requires about one disk write for each 10 files removed, whereas the traditional filesystem requires at least two writes for every file removed.

Soft Updates Requirements for fsck

As with the dual tracking of the true and effective link count, the changes needed to fsck became evident through operational experience. In a non-soft-updates filesystem implementation, file removal happens within a few milliseconds. Thus, there is a short period of time between the directory entry being removed and the inode being deallocated. If the system crashes during a bulk tree removal operation, there are usually no inodes lacking references from directory entries, though in rare instances there may be one or two. By contrast, in a system running with soft updates, many seconds may elapse between the times when the directory entry is deleted and the inode is deallocated. If the system crashes during a bulk tree removal operation, there are usually tens to hundreds of inodes lacking references from directory entries. Historically, fsck placed any unreferenced inodes into the lost+found directory. This action is reasonable if the filesystem has been damaged because of disk failure that results in the loss of one or more directories. However, it results in the incorrect action of stuffing the lost+found directory full of partially deleted files when running with soft updates. Thus, the fsck program was modified to check that a filesystem is running with soft updates and clear out, rather than save, unreferenced inodes (unless it has determined that unexpected damage has occurred to the filesystem, in which case the files are saved in lost+found).

A peripheral benefit of soft updates is that fsck can trust the allocation information in the bitmaps. Thus, it only needs to check the subset of inodes in the filesystem that the bitmaps show are in use. Although some of the inodes marked "in use" may be free, none of those marked "free" will ever be in use.

Performance of Soft Updates

We give only a cursory look at soft updates performance. For a detailed analysis, including comparisons with other techniques, see Ganger, McKusick et al. [2000].

We place the performance of soft updates in context by comparing it to the filesystem running without soft updates enabled (referred to below as "normal") that uses synchronous writes for update ordering and a filesystem mounted with the O_ASYNC option (referred to below as "asynchronous") that ignores all update dependencies. In asynchronous mode, all metadata updates are converted into delayed writes (see Section 6.6). Thus, the O_ASYNC data provides an upper bound on the performance of an update ordering scheme. As expected, we have found that soft updates eliminate almost all synchronous writes and usually allow the filesystem to achieve performance within 5 percent of the upper bound. Compared to using synchronous writes, file creation and removal performance increase by factors of 2 and 20, respectively. Generally, a filesystem running with soft updates tends to require 40 percent fewer disk writes and complete tasks 25 percent more quickly than when running without soft updates.

To provide a feeling for how the system performs in normal operation, we present measurements from three different system tasks. The first task is our "filesystem torture test." This test consists of 1000 runs of the Andrew benchmark, 1000 copy and removes of /etc with randomly selected pauses of 0 to 60 seconds between each copy and remove, and 500 executions of the find application from / with randomly selected pauses of 100 seconds between each run. The run of the torture test is shown in Table 8.3. The overall result is that asynchronous and soft updates require 42 percent fewer writes (with almost no synchronous writes) and have a 28 percent shorter running time. This result is particularly impressive when one considers that the finds and the pauses involve no update dependencies, and the Andrew benchmark is largely CPU bound.

Table 8.3. Running-time comparison for filesystem tests.

Filesystem

Disk Writes

Running

Configuration

Sync

Async

Time

Normal

1,459,147

487,031

27hr, 15min

Asynchronous

0

1,109,711

19hr, 43min

Soft Updates

6

1,113,686

19hr, 50min

The second test consists of building and installing the FreeBSD system. This task is a real-world example of a program development environment. The results are shown in Table 8.4. The overall result is that soft updates require 75 percent fewer writes and have a 21 percent shorter running time. Although soft updates do 30 percent more writes than asynchronous, the two result in the same running time.

Table 8.4. Running-time comparison for building FreeBSD.

Filesystem

Disk Writes

Running

Configuration

Sync

Async

Time

Normal

162,410

39,924

2hr, 12min

Asynchronous

0

38,262

1hr, 44min

Soft Updates

1124

48,850

1hr, 44min

The third test compares the performance of the central mail server for Berkeley Software Design, Inc., run with and without soft updates. The administrator was obviously unwilling to run it in asynchronous mode, since it is a production machine and people will not abide by losing their mail. Unlike the preceding tests that involve a single disk, the mail spool on this system is striped across three disks. The statistics were gathered by averaging the results from 30 days of nonweekend operation in each mode. The results for a 24-hour period are shown in Table 8.5. The normal filesystem averaged over 40 writes per second with a ratio of synchronous to asynchronous writes of 1:1. With soft updates, the write rate dropped to 12 per second, and the ratio of synchronous to asynchronous writes dropped to 1:8. For this real-world application, soft updates require 70 percent fewer writes, which triples the mail-handling capacity of the machine.

Table 8.5. Running-time comparison for a mail server.

Filesystem

Disk Writes

Configuration

Sync

Async

Normal

1,877,794

1,613,465

Soft Updates

118,102

946,519


8.7. Filesystem Snapshots

A filesystem snapshot is a frozen image of a filesystem at a given instant in time. Snapshots support several important features: the ability to provide backups of the filesystem at several times during the day, the ability to do reliable dumps of live filesystems, and (most important for soft updates) the ability to run a filesystem check program on an active system to reclaim lost blocks and inodes.

Creating a Filesystem Snapshot

Implementing snapshots has proven to be straightforward. Taking a snapshot entails the following steps:

1. A snapshot file is created to track later changes to the filesystem; a snapshot file is shown in Figure 8.28 (on page 350). This snapshot file is initialized to the size of the filesystem's partition, and its file block pointers are marked as zero, which means "not copied." A few strategic blocks are allocated, such as those holding copies of the superblock and cylinder group maps.

Figure 8.28. Structure of a snapshot file.

2. A preliminary pass is made over each of the cylinder groups to copy it to its preallocated backing block. Additionally, the block bitmap in each cylinder group is scanned to determine which blocks are free. For each free block that is found, the corresponding location in the snapshot file is marked with a distinguished block number (1) to show that the block is "not used." There is no need to copy those unused blocks if they are later allocated and written.

3. The filesystem is marked as "wanting to suspend." In this state, processes that wish to invoke system calls that will modify the filesystem are blocked from running, while processes that are already in progress on such system calls are permitted to finish them. These actions are enforced by inserting a gate at the top of every system call that can write to a filesystem. The set of gated system calls includes write, open (when creating or truncating), fhopen (when creating or truncating), mknod, mkfifo, link, symlink, unlink, chflags, fchflags, chmod, lchmod, fchmod, chown, lchown, fchown, utimes, lutimes, futimes, truncate, ftruncate, rename, mkdir, rmdir, fsync, sync, unmount, undelete, quotactl, revoke, and extattrctl. In addition gates must be added to pageout, ktrace, local-domain socket creation, and core dump creation. The gate tracks activity within a system call for each mounted filesystem. A gate has two purposes. The first is to suspend processes that want to enter the gated system call during periods that the filesystem that the process wants to modify is suspended. The second is to keep track of the number of processes that are running inside the gated system call for each mounted filesystem. When a process enters a gated system call, a counter in the mount structure for the filesystem that it wants to modify is incremented. When the process exits a gated system call, the counter is decremented.

4. The filesystem's status is changed from "wanting to suspend" to "fully suspended." This status change is done by allowing all system calls currently writing to the filesystem being suspended to finish. The transition to "fully suspended" is complete when the count of processes within gated system calls drops to zero.

5. The filesystem is synchronized to disk as if it were about to be unmounted.

6. Any cylinder groups that were modified after they were copied in step two are recopied to their preallocated backing block. Additionally, the block bitmap in each recopied cylinder group is rescanned to determine which blocks were changed. Newly allocated blocks are marked as "not copied," and newly freed blocks are marked as "not used." The details on how these modified cylinder groups are identified follows. The amount of space initially claimed by a snapshot is small, usually less than a tenth of one percent. Snapshot file space utilization is given in the snapshot performance subsection.

7. With the snapshot file in place, activity on the filesystem resumes. Any processes that were blocked at a gate are awakened and allowed to proceed with their system call.

8. Blocks that had been claimed by any snapshots that existed at the time that the current snapshot was taken are expunged from the new snapshot for reasons described below.

During steps 3 through 6, all write activity on the filesystem is suspended. Steps 3 and 4 complete in at most a few milliseconds. The time for step 5 is a function of the number of dirty pages in the kernel. It is bounded by the amount of memory that is dedicated to storing file pages. It is typically less than a second and is independent of the size of the filesystem. Typically step 6 needs to recopy only a few cylinder groups, so it also completes in less than a second.

The splitting of the bitmap copies between steps 2 and 6 is the way that we avoid having the suspend time be a function of the size of the filesystem. By making the primary copy pass while the filesystem is still active, and then having only a few cylinder groups in need of recopying after it has been suspended, we keep the suspend time down to a small and usually filesystem size independent time.

The details of the two-pass algorithm are as follows. Before starting the copy and scan of all the cylinder groups, the snapshot code allocates a "progress" bitmap whose size is equal to the number of cylinder groups in the filesystem. The purpose of the "progress" bitmap is to keep track of which cylinder groups have been scanned. Initially, all the bits in the "progress" map are cleared. The first pass is completed in step 2 before the filesystem is suspended. In this first pass, all the cylinder groups are scanned. When the cylinder group is read, its corresponding bit is set in the "progress" bitmap. The cylinder group is then copied, and its block map is consulted to update the snapshot file as described in step 2. Since the filesystem is still active, filesystem blocks may be allocated and freed while the cylinder groups are being scanned. Each time a cylinder group is updated because of a block being allocated or freed, its corresponding bit in the "progress" bitmap is cleared. Once this first pass over the cylinder groups is completed, the filesystem is "suspended."

Step 6 now becomes the second pass of the algorithm. The second pass need only identify and update the snapshot for any cylinder groups that were modified after it handled them in the first pass. The changed cylinder groups are identified by scanning the "progress" bitmap and rescanning any cylinder groups whose bits are zero. Although every bitmap would have to be reprocessed in the worst case, in practice only a few bitmaps need to be recopied and checked.

Maintaining a Filesystem Snapshot

Each time an existing block in the filesystem is modified, the filesystem checks whether that block was in use at the time that the snapshot was taken (i.e., it is not marked "not used"). If so, and if it has not already been copied (i.e., it is still marked "not copied"), a new block is allocated from among the "not used" blocks and placed in the snapshot file to replace the "not copied" entry. The previous contents of the block are copied to the newly allocated snapshot file block, and the modification to the original is then allowed to proceed. Whenever a file is removed, the snapshot code inspects each of the blocks being freed and claims any that were in use at the time of the snapshot. Those blocks marked "not used" are returned to the free list.

When a snapshot file is read, reads of blocks marked "not copied" return the contents of the corresponding block in the filesystem. Reads of blocks that have been copied return the contents in the copied block (e.g., the contents that were stored at that location in the filesystem at the time that the snapshot was taken). Writes to snapshot files are not permitted. When a snapshot file is no longer needed, it can be removed in the same way as any other file; its blocks are simply returned to the free list, and its inode is zeroed and returned to the free inode list.

Snapshots may live across reboots. When a snapshot file is created, the inode number of the snapshot file is recorded in the superblock. When a filesystem is mounted, the snapshot list is traversed and all the listed snapshots are activated. The only limit on the number of snapshots that may exist in a filesystem is the size of the array in the superblock that holds the list of snapshots. Currently, this array can hold up to 20 snapshots.

Multiple snapshot files can exist concurrently. As just described, earlier snapshot files would appear in later snapshots. If an earlier snapshot is removed, a later snapshot would claim its blocks rather than allowing them to be returned to the free list. This semantic means that it would be impossible to free any space on the filesystem except by removing the newest snapshot. To avoid this problem, the snapshot code goes through and expunges all earlier snapshots by changing its view of them to being zero-length files. With this technique, the freeing of an earlier snapshot releases the space held by that snapshot.

When a block is overwritten, all snapshots are given an opportunity to copy the block. A copy of the block is made for each snapshot in which the block resides. Overwrites typically occur only for inode and directory blocks. File data usually are not overwritten. Instead, a file will be truncated and then reallocated as it is rewritten. Thus, the slow and I/O intensive block copying is infrequent.

Deleted blocks are handled differently. The list of snapshots is consulted. When a snapshot is found in which the block is active ("not copied"), the deleted block is claimed by that snapshot. The traversal of the snapshot list is then terminated. Other snapshots for which the block is active are left with an entry of "not copied" for that block. The result is that when they access that location, they will still reference the deleted block. Since snapshots may not be modified, the block will not change. Since the block is claimed by a snapshot, it will not be allocated to another use. If the snapshot claiming the deleted block is deleted, the remaining snapshots will be given the opportunity to claim the block. Only when none of the remaining snapshots wants to claim the block (i.e., it is marked "not used" in all of them) will it be returned to the freelist.

Large Filesystem Snapshots

Creating and using a snapshot requires random access to the snapshot file. The creation of a snapshot requires the inspection and copying of all the cylinder group maps. Once in operation, every write operation to the filesystem must check whether the block being written needs to be copied. The information on whether a blocks needs to be copied is contained in the snapshot file metadata (its indirect blocks). Ideally, this metadata would be resident in the kernel memory throughout the lifetime of the snapshot. In FreeBSD, the entire physical memory on the machine can be used to cache file data pages if the memory is not needed for other purposes. Unfortunately, data pages associated with disks can only be cached in pages mapped into the kernel physical memory. Only about 10 megabytes of kernel memory is dedicated to such purposes. If we allow up to half of this space to be used for any single snapshot, the largest snapshot whose metadata that we can hold in memory is 11 Gbytes. Without help, such a tiny cache would be hopeless in trying to support a multiterabyte snapshot.

In an effort to support multiterabyte snapshots with the tiny metadata cache available, it is necessary to observe the access patterns on typical filesystems. The snapshot is only consulted for files that are being written. The filesystem is organized around cylinder groups that map small contiguous areas of the disk (see Section 8.9). Within a directory, the filesystem tries to allocate all the inodes and files in the same cylinder group. When moving between directories different cylinder groups are usually inspected. Thus, the widely random behavior occurs from movement between cylinder groups. Once file writing activity settles down into a cylinder group, only a small amount of snapshot metadata needs to be consulted. That metadata will easily fit in even the tiny kernel metadata cache. So the need is to find a way to avoid thrashing the cache when moving between cylinder groups.

The technique used to avoid thrashing when moving between cylinder groups is to build a look-aside table of all the blocks that were copied while the snapshot was made. This table lists the blocks associated with all the snapshot metadata blocks, the cylinder groups maps, the super block, and blocks that contain active inodes. When a copy-on-write fault occurs for a block, the first step is to consult this table. If the block is found in the table, then no further searching needs to be done in any of the snapshots. If the block is not found, then the metadata of each active snapshot on the filesystem must be consulted to see if a copy is needed. This table lookup saves time because it not only avoids faulting in metadata for widely scattered blocks, but it also avoids the need to consult potentially many snapshots.

Another problem with snapshots on large filesystems is that they aggravate existing deadlock problems. When there are multiple snapshots associated with a filesystem, they are kept in a list ordered from oldest to youngest. When a copy-on-write fault occurs, the list is traversed, letting each snapshot decide if it needs to copy the block that is about to be written. Originally, each snapshot inodc had its own lock. A deadlock could occur between two processes, each trying to do a write. Consider the example in Figure 8.29. It shows a filesystem with two snapshots: snap1 and snap2. Process A holds snapshot 1 locked, and process B holds snapshot 2 locked. Both snap1 and snap2 have decided that they need to allocate a new block in which to hold a copy of the block being written by the process that holds them locked. The writing of the new block in snapshot 1 will cause the kernel running in the context of process A to scan the list of snapshots that will get blocked at snapshot 2 because it is held locked by process B. Meanwhile, the writing of the new block in snapshot 2 will cause the kernel running in the context of process B to scan the list of snapshots that will get blocked at snapshot 1 because it is held locked by process A.

Figure 8.29. Snapshot deadlock scenario.

The resolution to the deadlock problem is to allocate a single lock that is used for all the snapshots on a filesystem. When a new snapshot is created, the kernel checks whether there are any other snapshots on the filesystem. If there are, the per-file lock associated with the new snapshot inode is released and replaced with the lock used for the other snapshots. With only a single lock, the access to the snapshots as a whole are serialized. Thus, in Figure 8.29, process B will hold the lock for all the snapshots and will be able to make the necessary checks and updates while process A will be held waiting. Once process B completes its scan, process A will be able to get access to all the snapshots and will be able to run successfully to completion. Because of the added serialization of the snapshot lookups, the look-aside table described earlier is important to ensure reasonable performance of snapshots. In gathering statistics on our running systems, we found that the look-aside table resolves nearly half of the snapshot copy-on-write lookups. Thus, we found that the look-aside table keeps the contention for the snapshot lock to a reasonable level.

Snapshot Performance

The experiments described in this section used the hardware and software configuration given in Table 8.6.

Table 8.6. Hardware and software configuration for snapshot testing.

Item

Configuration

Computer

Dual Processor using two Celeron 350MHz CPUs. The machine has 256 Mbyte of main memory.

Operating System

FreeBSD 5.0-current as of December 30, 2001.

I/O Controller

Adaptec 2940 Ultra2 SCSI adapter.

Disk

Two <IBM DDRS-39130D DClB> Fixed Direct Access SCSI-2 device, 80 Mbyte/sec transfers, Tagged Queuing Enabled, 8715 Mbyte, 17,850,000 512 byte sectors: 255 heads, 63 sectors per track, 1111 cylinders.

Small Filesystem

0.5 Gbyte, 8 Kbyte block, 1 Kbyte fragment, 90% full, 70874 files, initial snapshot size 0.392 Mbyte (0.08% of filesystem space).

Large Filesystem

7.7 Gbyte, 16 Kbyte block, 2 Kbyte fragment, 90% full, 520715 files, initial snapshot size 2.672 Mbyte (0.03% of filesystem space).

Load

Four continuously running simultaneous Andrew benchmarks that create a moderate amount of filesystem activity intermixed with periods of CPU intensive activity [Howard et al., 1988].

Table 8.7 shows the time to take a snapshot on an idle filesystem. The elapsed time to take a snapshot is proportional to the size of the filesystem being captured. However, nearly all the time to take a snapshot is spent in steps 1, 2, and 8. Because the filesystem permits other processes to modify the filesystem during steps 1, 2, and 8, this part of taking a snapshot does not interfere with normal system operation. The "suspend time" column shows the amount of real-time that processes are blocked from executing system calls that modify the filesystem. As Table 8.7 shows, the period during which write activity is suspended, and thus apparent to processes in the system, is short and does not increase proportionally to filesystem size.

Table 8.7. Snapshot times on an idle filesystem.

Filesystem Size

Elapsed Time

CPU Time

Suspend Time

0.5Gb

0.7 sec

0.1 sec

0.025 sec

7.7Gb

3.5 sec

0.4 sec

0.034 sec

Table 8.8 shows the times to snapshot a filesystem that has four active concurrent processes running. The elapsed time rises because the process taking the snapshot has to compete with the other processes for access to the filesystem. Note that the suspend time has risen slightly, but it is still insignificant and does not increase in proportion to the size of the filesystem under test. Instead, it is a function of the level of write activity present on the filesystem.

Table 8.8. Snapshot times on an active filesystem.

Filesystem Size

Elapsed Time

CPU Time

Suspend Time

0.5Gb

3.7 sec

0.1 sec

0.027 sec

7.7Gb

12.1 sec

0.4 sec

0.036 sec

Table 8.9 shows the times to remove a snapshot on an idle filesystem. The elapsed time to remove a snapshot is proportional to the size of the filesystem being captured. The filesystem does not need to be suspended to remove a snapshot.

Table 8.9. Snapshot removal time on an idle filesystem.

Filesystem Size

Elapsed Time

CPU Time

0.5Gb

0.5 sec

0.02 sec

7.7Gb

2.3 sec

0.09 sec

Table 8.10 shows the times to remove a snapshot on a filesystem that has four active concurrent processes running. The elapsed time rises because the process removing the snapshot has to compete with the other processes for access to the filesystem.

Table 8.10. Snapshot removal time on an active filesystem.

Filesystem Size

Elapsed Time

CPU Time

0.5Gb

1.4 sec

0.03 sec

7.7Gb

4.9 sec

0.10 sec

Background fsck

Traditionally, after an unclean system shutdown, the filesystem check program, fsck, has had to be run over all the inodes in a filesystem to ascertain which inodes and blocks are in use and to correct the bitmaps. This check is a painfully slow process that can delay the restart of a big server for an hour or more. The current implementation of soft updates guarantees the consistency of all filesystem resources, including the inode and block bitmaps. With soft updates, the only inconsistency that can arise in the filesystem (barring software bugs and media failures) is that some unreferenced blocks may not appear in the bitmaps and some inodes may have to have overly high link counts reduced. Thus, it is completely safe to begin using the filesystem after a crash without first running fsck. However, some filesystem space may be lost after each crash. Thus, there is value in having a version of fsck that can run in the background on an active filesystem to find and recover any lost blocks and adjust inodes with overly high link counts. A special case of the overly high link count is one that should be zero. Such an inode will be freed as part of reducing its link count to zero. This garbage collection task is less difficult than it might at first appear, since this version of fsck only needs to identify resources that are not in use and cannot be allocated or accessed by the running system.

With the addition of snapshots, the task becomes simple, requiring only minor modifications to the standard fsck. When run in background cleanup mode, fsck starts by taking a snapshot of the filesystem to be checked. Fsck then runs over the snapshot filesystem image doing its usual calculations just as in its normal operation. The only other change comes at the end of its run, when it wants to write out the updated versions of the bitmaps. Here, the modified fsck takes the set of blocks that it finds were in use at the time of the snapshot and removes this set from the set marked as in use at the time of the snapshot—the difference is the set of lost blocks. It also constructs the list of inodes whose counts need to be adjusted. Fsck then calls a new system call to notify the filesystem of the identified lost blocks so that it can replace them in its bitmaps. It also gives the set of inodes whose link counts need to be adjusted; those inodes whose link count is reduced to zero are truncated to zero length and freed. When fsck completes, it releases its snapshot. The complete details of how background fsck is implemented can be found in McKusick [2002; 2003].

User Visible Snapshots

Snapshots may be taken at any time. When taken every few hours during the day, they allow users to retrieve a file that they wrote several hours earlier and later deleted or overwrote by mistake. Snapshots are much more convenient to use than dump tapes and can be created much more frequently.

The snapshot described above creates a frozen image of a filesystem partition. To make that snapshot accessible to users through a traditional filesystem interface, the system administrator uses the vnode driver, vnd. The vnd driver takes a file as input and produces a character device interface to access it. The vnd character device can then be used as the input device for a standard mount command, allowing the snapshot to appear as a replica of the frozen filesystem at whatever location in the name space that the system administrator chooses to mount it.

Live Dumps

Once filesystem snapshots are available, it becomes possible to safely dump live filesystems. When dump notices that it is being asked to dump a mounted filesystem, it can simply take a snapshot of the filesystem and run over the snapshot instead of on the live filesystem. When dump completes, it releases the snapshot.


8.8. The Local Filestore

The final two sections of this chapter describe the organization and management of data on storage media. Historically FreeBSD provided three different filestore managers: the traditional Berkeley Fast Filesystem (FFS), the Log-Structured Filesystem, and the Memory-based Filesystem. FreeBSD 5.2 supports only FFS, although the other filesystems continue to be supported by NetBSD. The FFS filestore was designed on the assumption that file caches would be small and thus that files would need to be read often. It tries to place files likely to be accessed together in the same general location on the disk.

Overview of the Filestore

The operations defined for doing the datastore filesystem operations are shown in Table 8.11. These operators are fewer and semantically simpler than are those used for managing the name space.

Table 8.11. Datastore filesystem operations.

Operation Done

Operator Names

object creation and deletion

valloc, vfree

attribute update

Update

object read and write

vget, blkatoff, read, write, fsync

change in space allocation

truncate

There are two operators for allocating and freeing objects. The valloc operator creates a new object. The identity of the object is a number returned by the operator. The mapping of this number to a name is the responsibility of the namespace code. An object is freed by the vfree operator. The object to be freed is identified by only its number.

The attributes of an object are changed by the update operator. This layer does no interpretation of these attributes; they are simply fixed-size auxiliary data stored outside the main data area of the object. They are typically file attributes, such as the owner, group, permissions, and so on. Note that the extended attribute space is updated using the read and write interface as that interface is already prepared to read and write arbitrary length data to and from user-level processes.

There are five operators for manipulating existing objects. The vget operator retrieves an existing object from the filestore. The object is identified by its number and must have been created previously by valloc. The read operator copies data from an object to a location described by a uio structure. The blkatoff operator is similar to the read operator, except that the blkatoff operator simply returns a pointer to a kernel memory buffer with the requested data, instead of copying the data. This operator is designed to increase the efficiency of operations where the name-space code interprets the contents of an object (i.e., directories), instead of just returning the contents to a user process. The write operator copies data to an object from a location described by a uio structure. The fsync operator requests that all data associated with the object be moved to stable storage (usually by their all being written to disk). There is no need for an analog of blkatoff for writing, as the kernel can simply modify a buffer that it received from blkatoff, mark that buffer as dirty, and then do an fsync operation to have the buffer written back.

The final datastore operation is truncate. This operation changes the amount of space associated with an object. Historically, it could be used only to decrease the size of an object. In FreeBSD, it can be used both to increase and decrease the size of an object. When the size of a file is increased, a hole in the file is created. Usually, no additional disk space is allocated; the only change is to update the inode to reflect the larger file size. When read, holes are treated by the system as zero-valued bytes.

Each disk drive has one or more subdivisions, or partitions. Each such partition can contain only one filestore, and a filestore never spans multiple partitions. While a filesystem may use multiple disk partitions to do striping or RAID, the aggregation and management of the parts that make up the filesystem are managed by a lower-level driver in the kernel. The filesystem code always has the view of operating on a single contiguous partition.

The filestore is responsible for the management of the space within its disk partition. Within that space, its responsibility is the creation, storage, retrieval, and removal of files. It operates in a flat name space. When asked to create a new file, it allocates an inode for that file and returns the assigned number. The naming, access control, locking, and attribute manipulation for the file are all handled by the hierarchical filesystem-management layer above the filestore.

The filestore also handles the allocation of new blocks to files as the latter grow. Simple filesystem implementations, such as those used by early microcomputer systems, allocate files contiguously, one after the next, until the files reach the end of the disk. As files are removed, holes occur. To reuse the freed space, the system must compact the disk to move all the free space to the end. Files can be created only one at a time; for the size of a file other than the final one on the disk to be increased, the file must be copied to the end, then expanded.

As we saw in Section 8.2, each file in a filestore is described by an inode; the locations of its data blocks are given by the block pointers in its inode. Although the filestore may cluster the blocks of a file to improve I/O performance, the inode can reference blocks scattered anywhere throughout the partition. Thus, multiple files can be written simultaneously, and all the disk space can be used without the need for compaction.

The filestore implementation converts from the user abstraction of a file as an array of bytes to the structure imposed by the underlying physical medium. Consider a typical medium of a magnetic disk with fixed-sized sectoring. Although the user may wish to write a single byte to a file, the disk supports reading and writing only in multiples of sectors. Here, the system must read in the sector containing the byte to be modified, replace the affected byte, and write the sector back to the disk. This operation—converting random access to an array of bytes to reads and writes of disk sectors—is called block I/O.

First, the system breaks the user's request into a set of operations to be done on logical blocks of the file. Logical blocks describe block-sized pieces of a file. The system calculates the logical blocks by dividing the array of bytes into filestore-sized pieces. Thus, if a filestore's block size is 8192 bytes, then logical block 0 would contain bytes 0 to 8191, logical block 1 would contain bytes 8192 to 16,383, and so on.

The data in each logical block are stored in a physical block on the disk. A physical block is the location on the disk to which the system maps a logical block. A physical disk block is constructed from one or more contiguous sectors. For a disk with 512-byte sectors, an 8192-byte filestore block would be built up from 16 contiguous sectors. Although the contents of a logical block are contiguous on disk, the logical blocks of the file do not need to be laid out contiguously. The data structure used by the system to convert from logical blocks to physical blocks is described in Section 8.2.

User I/O to a File

Although the user may wish to write a single byte to a file, the disk hardware can read and write only in multiples of sectors. Hence, the system must arrange to read in the sector containing the byte to be modified, to replace the affected byte, and to write back the sector to the disk. This operation of converting random access to an array of bytes to reads and writes of disk sectors is known as block I/O.

Processes may read data in sizes smaller than a disk block. The first time that a small read is required from a particular disk block, the block will be transferred from the disk into a kernel buffer. Later reads of parts of the same block then require only copying from the kernel buffer to the memory of the user process. Multiple small writes are treated similarly. A buffer is allocated from the cache when the first write to a disk block is made, and later writes to part of the same block are then likely to require only copying into the kernel buffer, and no disk I/O.

In addition to providing the abstraction of arbitrary alignment of reads and writes, the block buffer cache reduces the number of disk I/O transfers required by filesystem accesses. Because system-parameter files, commands, and directories are read repeatedly, their data blocks are usually in the buffer cache when they are needed. Thus, the kernel does not need to read them from the disk every time that they are requested.

Figure 8.30 shows the flow of information and work required to access a file on the disk. The abstraction shown to the user is an array of bytes. These bytes are collectively described by a file descriptor that refers to some location in the array. The user can request a write operation on the file by presenting the system with a pointer to a buffer, with a request for some number of bytes to be written. As Figure 8.30 shows, the requested data do not need to be aligned with the beginning or end of a logical block. Further, the size of the request is not constrained to a single logical block. In the example shown, the user has requested data to be written to parts of logical blocks 1 and 2. Since the disk can transfer data only in multiples of sectors, the filestore must first arrange to read in the data for any part of the block that is to be left unchanged. The system must arrange an intermediate staging area for the transfer. This staging is done through one or more system buffers, described in Section 6.6.

Figure 8.30. The block I/O system.

In our example, the user wishes to modify data in logical blocks 1 and 2. The operation iterates over five steps:

1. Allocate a buffer.

2. Determine the location of the corresponding physical block on the disk.

3. Request the disk controller to read the contents of the physical block into the system buffer and wait for the transfer to complete.

4. Do a memory-to-memory copy from the beginning of the user's I/O buffer to the appropriate portion of the system buffer.

5. Write the block to the disk and continue without waiting for the transfer to complete.

If the user's request is incomplete, the process is repeated with the next logical block of the file. In our example, the system fetches logical block 2 of the file and is able to complete the user's request. Had an entire block been written, the system could have skipped step 3 and have simply written the data to the disk without first reading in the old contents. This incremental filling of the write request is transparent to the user's process because that process is blocked from running during the entire procedure. The filling is transparent to other processes; because the inode is locked during the process, any attempted access by any other process will be blocked until the write has completed.

If the system crashes while data for a particular block are in the cache but have not yet been written to disk, the filesystem on the disk will be incorrect and those data will be lost. The consistency of critical filesystem data are maintained using the techniques described in Section 8.6, but it is still possible to lose recently written application data. So that lost data are minimized, writes are forced for dirty buffer blocks 30 seconds after they are written. There is also a system call, fsync, that a process can use to force all dirty blocks of a single file to be written to disk immediately; this synchronization is useful for ensuring database consistency or before removing an editor backup file.


8.9. The Berkeley Fast Filesystem

A traditional UNIX filesystem is described by its superblock, which contains the basic parameters of the filesystem. These parameters include the number of data blocks in the filesystem, a count of the maximum number of files, and a pointer to the free list, which is a list of all the free blocks in the filesystem.

A 150-Mbyte traditional UNIX filesystem consists of 4 Mbyte of inodes followed by 146 Mbyte of data. That organization segregates the inode information from the data; thus, accessing a file normally incurs a long seek from the file's inode to its data. Files in a single directory typically are not allocated consecutive slots in the 4 Mbyte of inodes, causing many nonconsecutive disk blocks to be read when many inodes in a single directory are accessed.

The allocation of data blocks to files also is suboptimal. The traditional filesystem implementation uses a 512-byte physical block size. But the next sequential data block often is not on the same cylinder, so seeks between 512-byte data transfers are required frequently. This combination of small block size and scattered placement severely limits filesystem throughput.

The first work on the UNIX filesystem at Berkeley attempted to improve both the reliability and the throughput of the filesystem. The developers improved reliability by staging modifications to critical filesystem information so that the modifications could be either completed or repaired cleanly by a program after a crash [McKusick & Kowalski, 1994]. Doubling the block size of the filesystem improved the performance of the 4.0BSD filesystem by a factor of more than 2 when compared with the 3BSD filesystem. This doubling caused each disk transfer to access twice as many data blocks and eliminated the need for indirect blocks for many files. In the remainder of this section, we shall refer to the filesystem with these changes as the 3BSD filesystem.

The performance improvement in the 3BSD filesystem gave a strong indication that increasing the block size was a good method for improving throughput. Although the throughput had doubled, the 3BSD filesystem was still using only about 4 percent of the maximum disk throughput. The main problem was that the order of blocks on the free list quickly became scrambled as files were created and removed. Eventually, the free-list order became entirely random, causing files to have their blocks allocated randomly over the disk. This randomness forced a seek before every block access. Although the 3BSD filesystem provided transfer rates of up to 175 Kbyte per second when it was first created, the scrambling of the free list caused this rate to deteriorate to an average of 30 Kbyte per second after a few weeks of moderate use. There was no way of restoring the performance of a 3BSD filesystem except to recreate the system.

Organization of the Berkeley Fast Filesystem

The first version of the current BSD filesystem appeared in 4.2BSD [McKusick et al., 1984]. This version is still in use today as UFS1. In the FreeBSD filesystem organization (as in the 3BSD filesystem organization), each disk drive contains one or more filesystems. A FreeBSD filesystem is described by its superblock, located at the beginning of the filesystem's disk partition. Because the superblock contains critical data, it is replicated to protect against catastrophic loss. This replication is done when the filesystem is created. Since most of the superblock data do not change, the copies do not need to be referenced unless a disk failure causes the default superblock to be corrupted. The data in the superblock that does change include a few flags and some summary information that can easily be recreated if an alternative superblock needs to be used.

To allow support for filesystem fragments as small as a single 512-byte disk sector, the minimum size of a filesystem block is 4096 bytes. The block size can be any power of 2 greater than or equal to 4096. The block size is recorded in the filesystem's superblock, so it is possible for filesystems with different block sizes to be accessed simultaneously on the same system. The block size must be selected at the time that the filesystem is created; it cannot be changed subsequently without the filesystem being rebuilt.

The BSD filesystem organization divides a disk partition into one or more areas, each of which is called a cylinder group. Historically, a cylinder group comprised one or more consecutive cylinders on a disk. Although FreeBSD still uses the same data structure to describe cylinder groups, the practical definition of them has changed. When the filesystem was first designed, it could get an accurate view of the disk geometry including the cylinder and track boundaries and could accurately compute the rotational location of every sector. Modern disks hide this information, providing fictitious numbers of blocks per track, tracks per cylinder, and cylinders per disk. Indeed, in modern RAID arrays, the "disk" that is presented to the filesystem may really be composed from a collection of disks in the RAID array. While some research has been done to figure out the true geometry of a disk [Griffin et al., 2002; Lumb et al., 2002; Schindler et al., 2002], the complexity of using such information effectively is high. Modern disks have greater numbers of sectors per track on the outer part of the disk than the inner part that makes calculation of the rotational position of any given sector complex to calculate. So when the design for UFS2 was drawn up, we decided to get rid of all the rotational layout code found in UFS1 and simply assume that laying out files with numerically close block numbers (sequential being viewed as optimal) would give the best performance. Thus, the cylinder group structure is retained in UFS2, but it is used only as a convenient way to manage logically close groups of blocks. The rotational layout code had been disabled in UFS1 since the late 1980s, so as part of the code base cleanup it was removed entirely.

Each cylinder group must fit into a single filesystem block. When creating a new filesystem, the newfs utility calculates the maximum number of blocks that can be packed into a cylinder group map based on the filesystem block size. It then allocates the minimum number of cylinder groups needed to describe the filesystem. A filesystem with 16 Kbyte blocks typically has six cylinder groups per Gbyte.

Each cylinder group contains bookkeeping information that includes a redundant copy of the superblock, space for inodes, a bitmap describing available blocks in the cylinder group, and summary information describing the usage of data blocks within the cylinder group. The bitmap of available blocks in the cylinder group replaces the traditional filesystem's free list. For each cylinder group in UFS1, a static number of inodes is allocated at filesystem-creation time. The default policy is to allocate one inode per four filesystem fragments, with the expectation that this amount will be far more than will ever be needed. For each cylinder group in UFS2, the default is to reserve bitmap space to describe one inode per two filesystem fragments. In either type of filesystem, the default may be changed only at the time that the filesystem is created.

The rationale for using cylinder groups is to create clusters of inodes that are spread over the disk close to the blocks that they reference, instead of them all being located at the beginning of the disk. The filesystem attempts to allocate file blocks close to the inodes that describe them to avoid long seeks between getting the inodc and getting its associated data. Also, when the inodes are spread out, there is less chance of losing all of them in a single disk failure.

Although we decided to come up with a new on-disk inode format for UFS2, we chose not to change the format of the superblock, the cylinder group maps, or the directories. Additional information needed for the UFS2 superblock and cylinder groups is stored in spare fields of the UFS1 superblock and cylinder groups. Maintaining the same format for these data structures allows a single code base to be used for both UFS1 and UFS2. Because the only difference between the two filesystems is in the format of their inodes, code can dereference pointers to superblocks, cylinder groups, and directory entries without need of checking what type of filesystem is being accessed. To minimize conditional checking of code that references inodes, the on-disk inode is converted to a common in-core format when the inode is first read in from the disk and converted back to its on-disk format when it is written back. The effect of this decision is that there are only nine out of several hundred routines that are specific to UFS1 versus UFS2. The benefit of having a single code base for both filesystems is that it dramatically reduces the maintenance cost. Outside the nine filesystem format specific functions, fixing a bug in the code fixes it for both filesystem types. A common code base also means that as the symmetric multiprocessing support gets added, it only needs to be done once for the UFS family of filesystems.

Boot Blocks

The UFS1 filesystem reserved an 8-Kbyte space at the beginning of the filesystem in which to put a boot block. While this space seemed huge compared to the 1-Kbyte boot block that it replaced, over time it has gotten increasingly difficult to cram the needed boot code into this space. Consequently we decided to revisit the boot block size in UFS2.

The boot code has a list of locations to check for boot blocks. A boot block can be defined to start at any 8-Kbyte boundary. We set up an initial list with four possible boot block sizes: none, 8 Kbytes, 64 Kbytes, and 256 Kbytes. Each of these locations was selected for a particular purpose. Filesystems other than the root filesystem do not need to be bootable, so they can use a boot block size of zero. Also, filesystems on tiny media that need every block that they can get, such as floppy disks, can use a zero-size boot block. For architectures with simple boot blocks, the traditional UFS1 8-Kbyte boot block can be used. More typically the 64-Kbyte boot block is used (for example, on the PC architecture with its need to support booting from a myriad of busses and disk drivers).

We added the 256-Kbyte boot block in case some future architecture or application needs to set aside a particularly large boot area. This space reservation is not strictly necessary, since new sizes can be added to the list at any time, but it can take a long time before the updated list gets propagated to all the boot programs and loaders out on the existing systems. By adding the option for a huge boot area now, we can ensure it will be readily available should it be needed on short notice in the future.

An unexpected side effect of using a 64-Kbyte boot block for UFS2 is that if the partition had previously had a UFS1 filesystem on it, the superblock for the former UFS1 filesystem may not be overwritten. If an old version of fsck that does not first look for a UFS2 filesystem is run and finds the UFS1 superblock, it can incorrectly try to rebuild the UFS1 filesystem destroying the UFS2 filesystem in the process. So when building UFS2 filesystems, the newfs utility looks for old UFS1 superblocks and zeros them out.

Optimization of Storage Utilization

Data are laid out such that large blocks can be transferred in a single disk operation, greatly increasing filesystem throughput. A file in the new filesystem might be composed of 8192-byte data blocks, as compared to the 1024-byte blocks of the 3BSD filesystem; disk accesses would thus transfer up to 8 times as much information per disk transaction. In large files, several blocks can be allocated consecutively, so even larger data transfers are possible before a seek is required.

The main problem with larger blocks is that most BSD filesystems contain primarily small files. A uniformly large block size will waste space. Table 8.12 shows the effect of filesystem block size on the amount of wasted space in the filesystem. The measurements used to compute this table were collected from a survey of the Internet conducted in 1993 [Irlam, 1993]. The survey covered 12 million files residing on 1000 filesystems, with a total size of 250 Gbytes. The investigators found that the median file size was under 2048 bytes; the average file size was 22 Kbytes. The space wasted is calculated to be the percentage of disk space not containing user data. As the block size increases, the amount of space reserved for inodes decreases, but the amount of unused data space at the end of blocks rises quickly to an intolerable 29.4 percent waste, with a minimum allocation of 8192-byte filesystem blocks. More recent (proprietary) studies find similar results.

Table 8.12. Amount of space wasted as a function of block size.

Percentage Total Waste

Percentage Data Waste

Percentage Inode Waste

Organization

0.0

0.0

0.0

data only, no separation between files

1.1

1.1

0.0

data only, files start on 512-byte boundary

7.4

1.1

6.3

data + inodes, 512-byte block

8.8

2.5

6.3

data + inodes, 1024-byte block

11.7

5.4

6.3

data + inodes, 2048-byte block

15.4

12.3

3.1

data + inodes, 4096-byte block

29.4

27.8

1.6

data + inodes, 8192-byte block

62.0

61.2

0.8

data + inodes, 16,384-byte block

For large blocks to be used without significant waste, small files must be stored more efficiently. To increase space efficiency, the filesystem allows the division of a single filesystem block into one or more fragments. The fragment size is specified at the time that the filesystem is created; each filesystem block optionally can be broken into two, four, or eight fragments, each of which is addressable. The lower bound on the fragment size is constrained by the disk-sector size, which is typically 512 bytes. The block map associated with each cylinder group records the space available in a cylinder group in fragments; to determine whether a block is available, the system examines aligned fragments. Figure 8.31 shows a piece of a block map from a filesystem with 4096-byte blocks and 1024-byte fragments, hereinafter referred to as a 4096/1024 filesystem.

Figure 8.31. Example of the layout of blocks and fragments in a 4096/1024 filesystem. Each bit in the map records the status of a fragment; a "-" means that the fragment is in use, whereas a "1" means that the fragment is available for allocation. In this example, fragments 0 through 5, 10, and 11 are in use, whereas fragments 6 through 9 and 12 through 15 are free. Fragments of adjacent blocks cannot be used as a full block, even if they are large enough. In this example, fragments 6 through 9 cannot be allocated as a full block; only fragments 12 through 15 can be coalesced into a full block.

On a 4096/1024 filesystem, a file is represented by zero or more 4096-byte blocks of data, possibly plus a single fragmented block. If the system must fragment a block to obtain space for a few data, it makes the remaining fragments of the block available for allocation to other files. As an example, consider an 11,000-byte file stored on a 4096/1024 filesystem. This file would use two full-sized blocks and one three-fragment portion of another block. If no block with three aligned fragments were available at the time that the file was created, a full-sized block would be split, yielding the necessary fragments and a single unused fragment. This remaining fragment could be allocated to another file as needed.

Reading and Writing to a File

Having opened a file, a process can do reads or writes on it. The procedural path through the kernel is shown in Figure 8.32 (on page 368). If a read is requested, it is channeled through the ffs_read() routine. Ffs_read() is responsible for converting the read into one or more reads of logical file blocks. A logical block request is then handed off to ufs_bmap(). Ufs_bmap() is responsible for converting a logical block number to a physical block number by interpreting the direct and indirect block pointers in an inode. Ffs_read() requests the block I/O system to return a buffer filled with the contents of the disk block. If two or more logically sequential blocks are read from a file, the process is assumed to be reading the file sequentially. Here, ufs_bmap () returns two values: first, the disk address of the requested block and then the number of contiguous blocks that follow that block on disk. The requested block and the number of contiguous blocks that follow it are passed to the cluster() routine. If the file is being accessed sequentially, the cluster() routine will do a single large I/O on the entire range of sequential blocks. If the file is not being accessed sequentially (as determined by a seek to a different part of the file preceding the read), only the requested block or a subset of the cluster will be read. If the file has had a long series of sequential reads, or if the number of contiguous blocks is small, the system will issue one or more requests for read-ahead blocks in anticipation that the process will soon want those blocks. The details of block clustering are described at the end of this section.

Figure 8.32. Procedural interface to reading and writing.

[View full size image]

Each time that a process does a write system call, the system checks to see whether the size of the file has increased. A process may overwrite data in the middle of an existing file—in which case space would usually have been allocated already (unless the file contains a hole in that location). If the file needs to be extended, the request is rounded up to the next fragment size, and only that much space is allocated (see "Allocation Mechanisms" later in this section for the details of space allocation). The write system call is channeled through the ffs_write () routine. Ffs_write() is responsible for converting the write into one or more writes of logical file blocks. A logical block request is then handed off to ffs_balloc(). Ffs_balloc() is responsible for interpreting the direct and indirect block pointers in an inodc to find the location for the associated physical block pointer. If a disk block does not already exist, the ffs_alloc() routine is called to request a new block of the appropriate size. After calling chkdq() to ensure that the user has not exceeded his quota, the block is allocated, and the address of the new block is stored in the inode or indirect block. The address of the new or already-existing block is returned. Ffs_write() allocates a buffer to hold the contents of the block. The user's data are copied into the returned buffer, and the buffer is marked as dirty. If the buffer has been filled completely, it is passed to the cluster() routine. When a maximally sized cluster has been accumulated, a noncontiguous block is allocated, or a seek is done to another part of the file, and the accumulated blocks are grouped together into a single I/O operation that is queued to be written to the disk. If the buffer has not been filled completely, it is not considered immediately for writing. Instead, the buffer is held in the expectation that the process will soon want to add more data to it. It is not released until it is needed for some other block—that is, until it has reached the head of the free list or until a user process does an fsync system call. When a file acquires its first dirty block, it is placed on a 30-second timer queue. If it still has dirty blocks when the timer expires, all its dirty buffers are written. If it subsequently is written again, it will be returned to the 30-second timer queue.

Repeated small write requests may expand the file one fragment at a time. The problem with expanding a file one fragment at a time is that data may be copied many times as a fragmented block expands to a full block. Fragment reallocation can be minimized if the user process writes a full block at a time, except for a partial block at the end of the file. Since filesystems with different block sizes may reside on the same system, the filesystem interface provides application programs with the optimal size for a read or write. This facility is used by the standard I/O library that many application programs use and by certain system utilities, such as archivers and loaders, that do their own I/O management.

If the layout policies (described at the end of this section) are to be effective, a filesystem cannot be kept completely full. A parameter, termed the free-space reserve, gives the minimum percentage of filesystem blocks that should be kept free. If the number of free blocks drops below this level, only the superuser is allowed to allocate blocks. This parameter can be changed any time that the filesystem is unmounted. When the number of free blocks approaches zero, the filesystem throughput tends to be cut in half because the filesystem is unable to localize blocks in a file. If a filesystem's throughput drops because of overfilling, it can be restored by removal of files until the amount of free space once again reaches the minimum acceptable level. Users can restore locality to get faster access rates for files created during periods of little free space by copying the file to a new one and removing the original one when enough space is available.

Layout Policies

Each filesystem is parameterized so that it can be adapted to the characteristics of the application environment in which it is being used. These parameters are summarized in Table 8.13. In most situations, the default parameters work well, but in an environment with only a few large files or an environment with just a few huge directories, the performance can be enhanced by adjusting the layout parameters accordingly.

Table 8.13. Important parameters maintained by the filesystem.

Name

Default

Meaning

minfree

8%

minimum percentage of free space

avgfilesize

16K

expected average file size

filesperdir

64

expected number of files per directory

maxbpg

2048

maximum blocks per file in a cylinder group

maxcontig

8

maximum contiguous blocks that can be transfered in one I/O request

The filesystem layout policies are divided into two distinct parts. At the top level are global policies that use summary information to make decisions regarding the placement of new inodes and data blocks. These routines are responsible for deciding the placement of new directories and files. They also build contiguous block layouts and decide when to force a long seek to a new cylinder group because there is insufficient space left in the current cylinder group to do reasonable layouts. Below the global-policy routines are the local-allocation routines. These routines use a locally optimal scheme to lay out data blocks. The original intention was to bring out these decisions to user level so that they could be ignored or replaced by user processes. Thus, they are policies, rather than simple mechanisms.

Two methods for improving filesystem performance are to increase the locality of reference to minimize seek latency [Trivedi, 1980] and to improve the layout of data to make larger transfers possible [Nevalainen & Vesterinen, 1977]. The global layout policies try to improve performance by clustering related information. They cannot attempt to localize all data references but must instead try to spread unrelated data among different cylinder groups. If too much localization is attempted, the local cylinder group may run out of space, forcing further related data to be scattered to nonlocal cylinder groups. Taken to an extreme, total localization can result in a single huge cluster of data resembling the 3BSD filesystem. The global policies try to balance the two conflicting goals of localizing data that are concurrently accessed while spreading out unrelated data.

One allocatable resource is inodes. Inodes of files in the same directory frequently are accessed together. For example, the list-directory command, ls, may access the inode for each file in a directory. The inode layout policy tries to place all the inodes of files in a directory in the same cylinder group. To ensure that files are distributed throughout the filesystem, the system uses a different policy to allocate directory inodes. When a directory is being created in the root of the filesystem, it is placed in a cylinder group with a greater-than-average number of free blocks and inodes and with the smallest number of directories. The intent of this policy is to allow inode clustering to succeed most of the time. When a directory is created lower in the tree, it is placed in a cylinder group with with a greater-than-average number of free blocks and inodes near its parent directory. The intent of this policy is to reduce the distance tree-walking applications must seek as they move from directory to directory in a depth first search while still allowing inode clustering to succeed most of the time.

The filesystem allocates inodes within a cylinder group using a first-free strategy. Although this method allocates the inodes randomly within a cylinder group, it keeps the allocation down to the smallest number of inode blocks possible. Even when all the possible inodes in a cylinder group are allocated, they can be accessed with 10 to 20 disk transfers. This allocation strategy puts a small and constant upper bound on the number of disk transfers required to access the inodes for all the files in a directory. In contrast, the 3BSD filesystem typically requires one disk transfer to fetch the inode for each file in a directory.

The other major resource is the data blocks. Data blocks for a file typically are accessed together. The policy routines try to place data blocks for a file in the same cylinder group, preferably laid out contiguously. The problem with allocating all the data blocks in the same cylinder group is that large files quickly use up the available space, forcing a spillover to other areas. Further, using all the space causes future allocations for any file in the cylinder group also to spill to other areas. Ideally, none of the cylinder groups should ever become completely full. The heuristic chosen is to redirect block allocation to a different cylinder group after every few Mbyte of allocation. The spillover points are intended to force block allocation to be redirected when any file has used about 25 percent of the data blocks in a cylinder group. In day-to-day use, the heuristics appear to work well in minimizing the number of completely filled cylinder groups. Although this heuristic appears to benefit small files at the expense of the larger files, it really aids both file sizes. The small files are helped because there are nearly always blocks available in the cylinder group for them to use. The large files benefit because they are able to use the contiguous space available in the cylinder group and then to move on, leaving behind the blocks scattered around the cylinder group. Although these scattered blocks are fine for small files that need only a block or two, they slow down big files that are best stored on a single large group of blocks that can be read in a few disk revolutions.

The newly chosen cylinder group for block allocation is the next cylinder group that has a greater-than-average number of free blocks left. Although big files tend to be spread out over the disk, several Mbyte of data typically are accessible before a seek to a new cylinder group is necessary. Thus, the time to do one long seek is small compared to the time spent in the new cylinder group doing the I/O.

Allocation Mechanisms

The global-policy routines call local-allocation routines with requests for specific blocks. The local-allocation routines will always allocate the requested block if it is free; otherwise, they will allocate a free block of the requested size that is closest to the requested block. If the global layout policies had complete information, they could always request unused blocks, and the allocation routines would be reduced to simple bookkeeping. However, maintaining complete information is costly; thus, the global layout policy uses heuristics based on the partial information that is available.

If a requested block is not available, the local allocator uses a three-level allocation strategy:

  1. Use the next available block closest to the requested block in the same cylinder group.

  2. If the cylinder group is full, quadratically hash the cylinder-group number to choose another cylinder group in which to look for a free block. Quadratic hash is used because of its speed in finding unused slots in nearly full hash tables [Knuth, 1975]. Filesystems that are parameterized to maintain at least 8 percent free space rarely need to use this strategy. Filesystems used without free space typically have so few free blocks available that almost any allocation is random; the most important characteristic of the strategy used under such conditions is that it be fast.

  3. Apply an exhaustive search to all cylinder groups. This search is necessary because the quadratic rehash may not check all cylinder groups.

The task of managing block and fragment allocation is done by ffs_balloc(). If the file is being written and a block pointer is zero or points to a fragment that is too small to hold the additional data, ffs_balloc() calls the allocation routines to obtain a new block. If the file needs to be extended, one of two conditions exists:

  1. The file contains no fragmented blocks (and the final block in the file contains insufficient space to hold the new data). If space exists in a block already allocated, the space is filled with new data. If the remainder of the new data consists of more than a full block, a full block is allocated and the first full block of new data are written there. This process is repeated until less than a full block of new data remains. If the remaining new data to be written will fit in less than a full block, a block with the necessary number of fragments is located; otherwise, a full block is located. The remaining new data are written into the located space. However, to avoid excessive copying for slowly growing files, the filesystem allows only direct blocks of files to refer to fragments.

  2. The file contains one or more fragments (and the fragments contain insufficient space to hold the new data). If the size of the new data plus the size of the data already in the fragments exceeds the size of a full block, a new block is allocated. The contents of the fragments are copied to the beginning of the block, and the remainder of the block is filled with new data. The process then continues as in step 1. Otherwise, a set of fragments big enough to hold the data is located; if enough of the rest of the current block is free, the filesystem can avoid a copy by using that block. The contents of the existing fragments, appended with the new data, are written into the allocated space.

Ffs_balloc() is also responsible for allocating blocks to hold indirect pointers. It must also deal with the special case in which a process seeks past the end of a file and begins writing. Because of the constraint that only the final block of a file may be a fragment, ffs_balloc() must first ensure that any previous fragment has been upgraded to a full-sized block.

On completing a successful allocation, the allocation routines return the block or fragment number to be used; ffs_balloc() then updates the appropriate block pointer in the inode. Having allocated a block, the system is ready to allocate a buffer to hold the block's contents so that the block can be written to disk.

The procedural description of the allocation process is shown in Figure 8.33. Ffs_balloc() is the routine responsible for determining when a new block must be allocated. It first calls the layout-policy routine ffs_blkpref() to select the most desirable block based on the preference from the global-policy routines that were described earlier in this section. If a fragment has already been allocated and needs to be extended, ffs_balloc() calls ffs_realloccg(). If nothing has been allocated yet, ffs_balloc() calls ffs_alloc().

Figure 8.33. Procedural interface to block allocation.

Ffs_realloccg() first tries to extend the current fragment in place. Consider the sample block of an allocation map with two fragments allocated from it, shown in Figure 8.34 (on page 374). The first fragment can be extended from a size 2 fragment to a size 3 or a size 4 fragment, since the two adjacent fragments are unused. The second fragment cannot be extended, as it occupies the end of the block, and fragments are not allowed to span blocks. If ffs_realloccg() is able to expand the current fragment in place, the map is updated appropriately and it returns. If the fragment cannot be extended, ffs_realloccg() calls the ffs_alloc() routine to get a new fragment. The old fragment is copied to the beginning of the new fragment, and the old fragment is freed.

Figure 8.34. Sample block with two allocated fragments.

The bookkeeping tasks of allocation are handled by ffs_alloc(). It first verifies that a block is available in the desired cylinder group by checking the filesystem summary information. If the summary information shows that the cylinder group is full, ffs_alloc() quadratically rehashes through the summary information looking for a cylinder group with free space. Having found a cylinder group with space, ffs_alloc() calls either the fragment-allocation routine or the block-allocation routine to acquire a fragment or block.

The block-allocation routine is given a preferred block. If that block is available, it is returned. If the block is unavailable, the allocation routine tries to find another block in the same cylinder group that is close to the requested block. It looks for an available block by scanning forward through the free-block map, starting from the requested location until it finds an available block.

The fragment-allocation routine is given a preferred fragment. If that fragment is available, it is returned. If the requested fragment is not available, and the filesystem is configured to optimize for space utilization, the filesystem uses a best-fit strategy for fragment allocation. The fragment-allocation routine checks the cylinder-group summary information, starting with the entry for the desired size, and scanning larger sizes until an available fragment is found. If there are no fragments of the appropriate size or larger, then a full-sized block is allocated and is broken up.

If an appropriate-sized fragment is listed in the fragment summary, then the allocation routine expects to find it in the allocation map. To speed up the process of scanning the potentially large allocation map, the filesystem uses a table-driven algorithm. Each byte in the map is treated as an index into a fragment-descriptor table. Each entry in the fragment-descriptor table describes the fragments that are free for that corresponding map entry. Thus, by doing a logical AND with the bit corresponding to the desired fragment size, the allocator can determine quickly whether the desired fragment is contained within a given allocation-map entry. As an example, consider the entry from an allocation map for the 8192/1024 filesystem shown in Figure 8.35. The map entry shown has already been fragmented, with a single fragment allocated at the beginning and a size 2 fragment allocated in the middle. Remaining unused is another size 2 fragment and a size 3 fragment. Thus, if we lookup entry 115 in the fragment table, we find the entry shown in Figure 8.36. If we were looking for a size 3 fragment, we would inspect the third bit and find that we had been successful; if we were looking for a size 4 fragment, we would inspect the fourth bit and find that we needed to continue. The C code that implements this algorithm is as follows:

for (i = 0; i < MAPSIZE; i++)
    if (fragtbl [allocmap[i] ] & (1 << (size - 1)) )
        break;

Figure 8.35. Map entry for an 8192/1024 filesystem.

Figure 8.36. Fragment-table entry for entry 115.

Using a best-fit policy has the benefit of minimizing disk fragmentation; however, it has the undesirable property that it maximizes the number of fragment-to-fragment copies that must be made when a process writes a file in many small pieces. To avoid this behavior, the system can configure filesystems to optimize for time rather than for space. The first time that a process does a small write on a filesystem configured for time optimization, it is allocated a best-fit fragment. On the second small write, however, a full-sized block is allocated, with the unused portion being freed. Later small writes are able to extend the fragment in place, rather than requiring additional copy operations. Under certain circumstances, this policy can cause the disk to become heavily fragmented. The system tracks this condition and automatically reverts to optimizing for space if the percentage of fragmentation reaches one-half of the minimum free-space limit.

Block Clustering

Most machines running FreeBSD do not have separate I/O processors. The main CPU must take an interrupt after each disk I/O operation; if there is more disk I/O to be done, it must select the next buffer to be transferred and must start the operation on that buffer. Before the advent of track-caching controllers, the filesystem obtained its highest throughput by leaving a gap after each block to allow time for the next I/O operation to be scheduled. If the blocks were laid out without a gap, the throughput would suffer because the disk would have to rotate nearly an entire revolution to pick up the start of the next block.

Track-caching controllers have a large buffer in the controller that continues to accumulate the data coming in from the disk even after the requested data have been received. If the next request is for the immediately following block, the controller will already have most of the block in its buffer, so it will not have to wait a revolution to pick up the block. Thus, for the purposes of reading, it is possible to nearly double the throughput of the filesystem by laying out the files contiguously rather than leaving gaps after each block.

Unfortunately, the track cache is less useful for writing. Because the kernel does not provide the next data block until the previous one completes, there is still a delay during which the controller does not have the data to write, and it ends up waiting a revolution to get back to the beginning of the next block. One solution to this problem is to have the controller give its completion interrupt after it has copied the data into its cache, but before it has finished writing them. This early interrupt gives the CPU time to request the next I/O before the previous one completes, thus providing a continuous stream of data to write to the disk.

This approach has one seriously negative side effect. When the I/O completion interrupt is delivered, the kernel expects the data to be on stable store. Filesystem integrity and user applications using the fsync system call depend on these semantics. These semantics will be violated if the power fails after the I/O completion interrupt but before the data are written to disk. Some vendors eliminate this problem by using nonvolatile memory for the controller cache and providing microcode restart after power fail to determine which operations need to be completed. Because this option is expensive, few controllers provide this functionality.

Newer disks resolve this problem with a technique called tag queueing. With tag queueing, each request passed to the disk driver is assigned a unique numeric tag. Most disk controllers supporting tag queueing will accept at least 16 pending I/O requests. After each request is finished, the tag of the completed request is returned as part of the completion interrupt. If several contiguous blocks are presented to the disk controller, it can begin work on the next one while the tag for the previous one is being returned. Thus, tag queueing allows applications to be accurately notified when their data has reached stable store without incurring the penalty of lost disk revolutions when writing contiguous blocks.

To maximize throughput on systems without tag queueing or nonvolatile controller memory, the FreeBSD system implements I/O clustering. Clustering helps improve performance on all systems by reducing the number of I/O requests through the aggregation of many small requests into a smaller number of big ones. Clustering was first done by Santa Cruz Operations [Peacock, 1988] and Sun Microsystems [McVoy & Kleiman, 1991]; the idea was later adapted to 4.4BSD and thus to FreeBSD [Seltzer et al., 1993]. As a file is being written, the allocation routines try to allocate up to maxcontig (typically 128 Kbyte) of data in contiguous disk blocks. Instead of the buffers holding these blocks being written as they are filled, their output is delayed. The cluster is completed when the limit of maxcontig of data is reached, the file is closed, or the cluster cannot grow because the next sequential block on the disk is already in use by another file. If the cluster size is limited by a previous allocation to another file, the filesystem is notified and is given the opportunity to find a larger set of contiguous blocks into which the cluster may be placed. If the reallocation is successful, the cluster continues to grow. When the cluster is complete, the buffers making up the cluster of blocks are aggregated and passed to the disk controller as a single I/O request. The data can then be streamed out to the disk in a single uninterrupted transfer.

A similar scheme is used for reading. If the ffs_read() discovers that a file is being read sequentially, it inspects the number of contiguous blocks returned by ufs_bmap() to look for clusters of contiguously allocated blocks. It then allocates a set of buffers big enough to hold the contiguous set of blocks and passes them to the disk controller as a single I/O request. The I/O can then be done in one operation. Although read clustering is not needed when track-caching controllers are available, it reduces the interrupt load from systems that have them, and it speeds low-cost systems that do not have them.

For clustering to be effective, the filesystem must be able to allocate large clusters of contiguous blocks to files. If the filesystem always tried to begin allocation for a file at the beginning of a large set of contiguous blocks, it would soon use up its contiguous space. Instead, it uses an algorithm similar to that used for the management of fragments. Initially, file blocks are allocated via the standard algorithm described in the previous two subsections. Reallocation is invoked when the standard algorithm does not result in a contiguous allocation. The reallocation code searches a cluster map that summarizes the available clusters of blocks in the cylinder group. It allocates the first free cluster that is large enough to hold the file and then moves the file to this contiguous space. This process continues until the current allocation has grown to a size equal to the maximum permissible I/O operation (maxcontig). At that point, the I/O is done, and the process of allocating space begins again.

Unlike fragment reallocation, block reallocation to different clusters of blocks does not require extra I/O or memory-to-memory copying. The data to be written are held in delayed write buffers. Within that buffer is the disk location to which the data are to be written. When the block cluster is relocated, it takes little time to walk the list of buffers in the cluster and to change the disk addresses to which they are to be written. When the I/O occurs, the final destination has been selected and will not change.

To speed the operation of finding clusters of blocks, the filesystem maintains a cluster map with 1 bit per block (in addition to the map with 1 bit per fragment). It also has summary information showing how many sets of blocks there are for each possible cluster size. The summary information allows it to avoid looking for cluster sizes that do not exist. The cluster map is used because it is faster to scan than is the much larger fragment bitmap. The size of the map is important because the map must be scanned bit by bit. Unlike fragments, clusters of blocks are not constrained to be aligned within the map. Thus, the table-lookup optimization done for fragments cannot be used for lookup of clusters.

The filesystem relies on the allocation of contiguous blocks to achieve high levels of performance. The fragmentation of free space may increase with time or with filesystem utilization. This fragmentation can degrade performance as the filesystem ages. The effects of utilization and aging were measured on over 50 filesystems at Harvard University. The measured filesystems ranged in age since initial creation from one to three years. The fragmentation of free space on most of the measured filesystems caused performance to degrade no more than 10 percent from that of a newly created empty filesystem. The most severe degradation measured was 30 percent on a highly active filesystem that had many small files and was used to spool USENET news [Seltzer et al., 1995].

Extent-based Allocation

Figure 8.37. Alternative file metadata representations.

With the addition of dynamic block reallocation in the early 1990s [Seltzer & Smith, 1996], the UFS1 filesystem has had the ability to allocate most files contiguously on the disk. The metadata describing a large file consists of indirect blocks with long runs of sequential block numbers, see Figure 8.37(a). For quick access while a file is active, the kernel tries to keep all a file's metadata in memory. With UFS2 the space required to hold the metadata for a file is doubled as every block pointer grows from 32 bits to 64 bits. To provide a more compact representation, many filesystems use an extent-based representation. A typical extent-based representation uses pairs of block numbers and lengths. Figure 8.37(b) represents the same set of block numbers as Figure 8.37(a) in an extent-based format. If the file can be laid out nearly contiguously, this representation provides a compact description. However, randomly or slowly written files can end up with many noncontiguous block allocations, which will produce a representation that requires more space than the one used by UFS1. This representation also has the drawback that it can require much computation to do random access to the file, since the block number needs to be computed by adding up the sizes starting from the beginning of the file until the desired seek offset is reached.

To gain most of the efficiencies of extents without the random access inefficiencies, UFS2 has added a field to the inode that will allow that inode to use a larger block size. Small, slowly growing, or sparse files set this value to the regular filesystem block size and represent their data in the traditional way shown in Figure 8.37(a). However, when the filesystem detects a large, dense file, it can set this inode-block-size field to a value 2 to 16 times the filesystem block size. Figure 8.37(c) represents the same set of block numbers as Figure 8.37(a), with the inode-block-size field set to 4 times the filesystem block size. Each block pointer references a piece of disk storage that is four times larger, which reduces the metadata storage requirement by 75 percent. Since every block pointer other than possibly the last one references an equal-sized block, computation of random access offsets is just as fast as in the traditional metadata representation. Unlike the traditional extent-based representation that can double the metadata space requirement for certain datasets, this representation will always result in less space dedicated to metadata.

The drawback to this approach is that once a file has committed to using a larger block size, it can only use blocks of that size. If the filesystem runs out of big blocks, then the file can no longer grow, and either the application will get an "out-of-space" error or the filesystem has to recreate the metadata with the standard filesystem block size. The current plan is to write the code to recreate the metadata. While recreating the metadata usually will cause a long pause, we expect that condition to be rare and not a noticeable problem in real use.


Exercises

8.1 What are the seven classes of operations handled by the hierarchical filesystem?

8.2 What is the purpose of the inode data structure?

8.3 How does the system select an inode for replacement when a new inode must be brought in from disk?

8.4 Why are directory entries not allowed to span chunks?

8.5 Describe the steps involved in looking up a pathname component.

8.6 Why are hard links not permitted to span filesystems?

8.7 Describe how the interpretation of a symbolic link containing an absolute pathname is different from that of a symbolic link containing a relative pathname.

8.8 Explain why unprivileged users are not permitted to make hard links to directories but are permitted to make symbolic links to directories.

8.9 How can hard links be used to gain access to files that could not be accessed if a symbolic link were used instead?

8.10 How does the system recognize loops caused by symbolic links? Suggest an alternative scheme for doing loop detection.

8.11 How do quotas differ from the file-size resource limits described in Section 3.8?

8.12 How does the kernel determine whether a file has an associated quota?

8.13 Draw a picture showing the effect of processing an exclusive-lock request by process 1 on bytes 7 to 10 to the lock list shown in Figure 8.15. Which of the overlap cases of Figure 8.14 apply to this example?

8.14 In the absence of soft updates, which three FFS operations must be done synchronously to ensure that the filesystem can always be recovered deterministically after a crash (barring unrecoverable hardware errors)?

8.15 What are the guarantees made by the fsync system call?

8.16 Name the three ordering constraints that must be maintained when a file is removed. Describe how soft updates maintains this ordering.

8.17 Give three uses for a filesystem snapshot.

8.18 Describe the eight steps needed to take a filesystem snapshot.

8.19 What are the three states that a block may have in a snapshot? Describe the actions taken by a snapshot for each of these states when a write occurs. Describe the actions taken by a snapshot for each of these states when a block is released.

8.20 What are the four classes of operations handled by the datastore filesystem?

8.21 Under what circumstances can a write request avoid reading a block from the disk?

8.22 What is the difference between a logical block and a physical block? Why is this distinction important?

8.23 Give two reasons why increasing the basic block size in the old filesystem from 512 bytes to 1024 bytes more than doubled the system's throughput.

8.24 How many blocks and fragments are allocated to a 31,200-byte file on a FFS with 4096-byte blocks and 1024-byte fragments? How many blocks and fragments are allocated to this file on a FFS with 4096-byte blocks and 512-byte fragments? Also, answer these two questions assuming that an inode had only 6 direct block pointers instead of 12.

8.25 Explain why the FFS maintains a 5 to 10 percent reserve of free space. What problems would arise if the free-space reserve were set to zero?

8.26 What is a quadratic hash? Describe for what it is used in the FFS, and why it is used for that purpose.

8.27 Why are the allocation policies for inodes different from those for data blocks?

8.28 Under what circumstances does block clustering provide benefits that cannot be obtained with a disk-track cache?

*8.29 Give an example where the file-locking implementation is unable to detect a potential deadlock.

*8.30 What problems would arise if files had to be allocated in a single contiguous piece of the disk? Consider the problems created by multiple processes, random access, and files with holes.

**8.31 Design a system that allows the security level of the system to be lowered while the system is still running in multiuser mode.

**8.32 Inodes could be allocated dynamically as part of a directory entry. Instead, the inode allocation region is reserved when the filesystem is created. Why is the latter approach used?


References

Apple, 2003. Apple, "Mac OS X Essentials, Chapter Filesystem, Section 12 Resource Forks," http://developer.apple.com/techpubs/macosx/Essentials/SystemOverview/FileSystem/chapter_9_section_12.html, 2003.

Best & Kleikamp, 2003. S. Best & D. Kleikamp, "How the Journaled File System handles the on-disk layout," http://www-106.ibm.com/developerworks/linux/library/l-jfslayout/, 2003.

Chamberlin & Astrahan, 1981. D. Chamberlin & M. Astrahan, "A History and Evaluation of System R," Communications of the ACM, vol. 24, no. 10, pp. 632-646, 1981.

Chutani et al., 1992. S. Chutani, O. Anderson, M. Kazar, W. Mason, & R. Sidebotham, "The Episode File System," USENIX Association Conference Proceedings, pp. 43-59, January 1992.

Dowse & Malone, 2002. I. Dowse & D. Malone, "Recent Filesystem Optimizations on FreeBSD," Proceedings of the Freenix Track at the 2002 Usenix Annual Technical Conference, pp. 245-258, June 2002.

Elz, 1984. K. R. Elz, "Resource Controls, Privileges, and Other MUSH," USENIX Association Conference Proceedings, pp. 183-191, June 1984.

Ganger, McKusick et al., 2000. G. Ganger, M. K. McKusick, C. Soules, & Y. Patt, "Soft Updates: A Solution to the Metadata Update Problem in File Systems," ACM Transactions on Computer Systems, vol. 18, no. 2, pp. 127-153, May 2000.

Ganger & Patt, 1994. G. Ganger & Y. Patt, "Metadata Update Performance in File Systems," USENIX Symposium on Operating Systems Design and Implementation, pp. 49-60, November 1994.

Griffin et al., 2002. J. L. Griffin, J. Schindler, S. W. Schlosser, J. S. Bucy, & G. R. Ganger, "Timing-accurate Storage Emulation," Proceedings of the Usenix Conference on File and Storage Technologies, pp. 75-88, January 2002.

Hagmann, 1987. R. Hagmann, "Reimplementing the Cedar File System Using Logging and Group Commit," ACM Symposium on Operating Systems Principles, pp. 155-162, November 1987.

Howard et al., 1988. J. Howard, M. Kazar, S. Menees, D. Nichols, M. Satyanarayanan, R. Sidebotham, & M. West, "Scale and Performance in a Distributed File System," ACM Transactions on Computer Systems, vol. 6, no. 1, pp. 51-81, Association for Computing Machinery, February 1988.

Irlam, 1993. G. Irlam, Unix File Size Survey—1993, http://www.base.com/gordoni/ufs93.html, email:<gordoni@home.base.com>, November 1993.

Knuth, 1975. D. Knuth, The Art of Computer Programming, Volume 3—Sorting and Searching, pp. 506-549, Addison-Wesley, Reading, MA, 1975.

Lumb et al., 2002. C. R. Lumb, J. Schindler, & G. R. Ganger, "Freeblock Scheduling Outside of Disk Firmware," Proceedings of the Usenix Conference on File and Storage Technologies, pp. 275-288, January 2002.

McKusick, 2002. M. K. McKusick, "Running Fsck in the Background," Proceedings of the BSDCon 2002 Conference, pp. 55-64, February 2002.

McKusick, 2003. M. K. McKusick, "Enhancements to the Fast Filesystem to Support Multi-Terabyte Storage Systems," Proceedings of the BSDCon 2003 Conference, pp. 79-90, September 2003.

McKusick et al., 1984. M. K. McKusick, W. N. Joy, S. J. Leffler, & R. S. Fabry, "A Fast File System for UNIX," ACM Transactions on Computer Systems, vol. 2, no. 3, pp. 181-197, Association for Computing Machinery, August 1984.

McKusick & Kowalski, 1994. M. K. McKusick & T. J. Kowalski, "Fsck: The UNIX File System Check Program," in 4.4BSD System Manager's Manual, pp. 3:1-21, O'Reilly & Associates, Inc., Sebastopol, CA, 1994.

McVoy & Kleiman, 1991. L. McVoy & S. Kleiman, "Extent-Like Performance from a Unix File System," USENIX Association Conference Proceedings, pp. 33-44, January 1991.

Moran et al., 1990. J. Moran, R. Sandberg, D. Coleman, J. Kepecs, & B. Lyon, "Breaking Through the NFS Performance Barrier," Proceedings of the Spring 1990 European UNIX Users Group Conference, pp. 199-206, April 1990.

Nevalainen & Vesterinen, 1977. O. Nevalainen & M. Vesterinen, "Determining Blocking Factors for Sequential Files by Heuristic Methods," The Computer Journal, vol. 20, no. 3, pp. 245-247, August 1977.

Ousterhout, 1990. J. Ousterhout, "Why Aren't Operating Systems Getting Faster as Fast as Hardware?," Summer USENIX Conference, pp. 247-256, June 1990.

Peacock, 1988. J. Peacock, "The Counterpoint Fast File System," USENIX Association Conference Proceedings, pp. 243-249, January 1988.

Phillips, 2001. D. Phillips, "A Directory Index for Ext2," Proceedings of the Usenix Fifth Annual Linux Showcase and Conference, November 2001.

Reiser, 2001. H. Reiser, "The Reiser File System," http://www.namesys.com/res_whol.shtml, January 2001.

Rhodes, 2003. T. Rhodes, "FreeBSD Handbook, Chapter 3 , Section File System Access Control Lists," http://www.FreeBSD.org/doc/en_US.IS08859-1/books/handbook/fs-acl.html, 2003.

Rosenblum & Ousterhout, 1992. M. Rosenblum & J. Ousterhout, "The Design and Implementation of a Log-Structured File System," ACM Transactions on Computer Systems, vol. 10, no. 1, pp. 26-52, Association for Computing Machinery, February 1992.

Schindler et al., 2002. J. Schindler, J. L. Griffin, C. R. Lumb, & G. R. Ganger, "Track-aligned Extents: Matching Access Patterns to Disk Drive Characteristics," Proceedings of the Usenix Conference on File and Storage Technologies, pp. 259-274, January 2002.

Seltzer et al., 1993. M. Seltzer, K. Bostic, M. K. McKusick, & C. Staelin, "An Implementation of a Log-Structured File System for UNIX," USENIX Association Conference Proceedings, pp. 307-326, January 1993.

Seltzer & Smith, 1996. M. Seltzer & K. Smith, "A Comparison of FFS Disk Allocation Algorithms," Winter USENIX Conference, pp. 15-25, January 1996.

Seltzer et al., 1995. M. Seltzer, K. Smith, H. Balakrishnan, J. Chang, S. McMains, & V. Padmanabhan, "File System Logging Versus Clustering: A Performance Comparison," USENIX Association Conference Proceedings, pp. 249-264, January 1995.

Stonebraker, 1987. M. Stonebraker, "The Design of the POSTGRES Storage System," Very Large DataBase Conference, pp. 289-300, 1987.

Sweeney et al., 1996. A. Sweeney, D. Doucette, C. Anderson, W. Hu, M. Nishimoto, & G. Peck, "Scalability in the XFS File System," Proceedings of the 1996 Usenix Annual Technical Conference, pp. 1-14, January 1996.

Trivedi, 1980. K. Trivedi, "Optimal Selection of CPU Speed, Device Capabilities, and File Assignments," Journal of the ACM, vol. 27, no. 3, pp. 457-473, July 1980.

Watson, 2000. R. Watson, "Introducing Supporting Infrastructure for Trusted Operating System Support in FreeBSD," Proceedings of the BSDCon 2000 Conference, September 2000.

Watson, 2001. R. Watson, "TrustedBSD: Adding Trusted Operating System Features to FreeBSD," Proceedings of the Freenix Track at the 2001 Usenix Annual Technical Conference, pp. 15-28, June 2001.

Watson et al., 2003. R. Watson, W. Morrison, C. Vance, & B. Feldman, "The TrustedBSD MAC Framework: Extensible Kernel Access Control for FreeBSD 5.0," Proceedings of the Freenix Track at the 2003 Usenix Annual Technical Conference, pp. 285-296, June 2003.

Wu & Zwaenepoel, 1994. M. Wu & W. Zwaenepoel, "eNVy: A Non-Volatile, Main Memory Storage System," International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pp. 86-97, October 1994.


Chapter 9. The Network Filesystem

This chapter is divided into three main sections. The first gives a brief history of remote filesystems. The second describes the client and server halves of NFS and the mechanics of how they operate. The final section describes the techniques needed to provide reasonable performance for remote filesystems in general, and NFS in particular.


9.1. History and Overview

When networking first became widely available in 4.2BSD, users who wanted to share files all had to log in across the net to a central machine on which the shared files were located. These central machines quickly became far more loaded than the user's local machine, so demand quickly grew for a convenient way to share files on several machines at once. The most easily understood sharing model is one that allows a server machine to export its filesystems to one or more client machines. The clients can then import these filesystems and present them to the user as though they were just another local filesystem.

Numerous remote-filesystem protocol designs and protocols were proposed and implemented. The implementations were attempted at all levels of the kernel. Remote access at the top of the kernel resulted in semantics that nearly matched the local filesystem but had terrible performance. Remote access at the bottom of the kernel resulted in awful semantics but great performance. Modern systems place the remote access in the middle of the kernel at the vnode layer. This level gives reasonable performance and acceptable semantics.

An early remote filesystem, UNIX United, was implemented near the top of the kernel at the system-call dispatch level. It checked for file descriptors representing remote files and sent requests on those descriptors off to the server. No caching was done on the client machine. The lack of caching resulted in slow performance but in semantics nearly identical to a local filesystem. Because the current directory and executing files are referenced internally by vnodes rather than by descriptors, UNIX United did not allow users to change directory into a remote filesystem and could not execute files from a remote filesystem without first copying the files to a local filesystem.

At the opposite extreme was Sun Microsystem's network disk, implemented near the bottom of the kernel at the device-driver level. Here, the client's entire filesystem and buffering code was used. Just as in the local filesystem, recently read blocks from the disk were stored in the page cache. Only when a file access requested a block that was not already in the cache would the client send a request for the needed physical disk block to the server. The performance was excellent because the cache serviced most of the file-access requests just as it does for the local filesystem. Unfortunately, the semantics suffered because of incoherency between the client and server caches. Changes made on the server would not be seen by the client, and vice versa. As a result, the network disk could be used only by a single client or as a read-only filesystem.

The first remote filesystem shipped with System V was RFS [Rifkin et al., 1986]. Although it had excellent UNIX semantics, its performance was poor, so it was seldom used. Research at Carnegie-Mellon lead to the Andrew filesystem [Howard, 1988]. The Andrew filesystem was commercialized by Transarc and eventually became part of the Distributed Computing Environment promulgated by the Open Software Foundation and was supported by many vendors. It is designed to handle widely distributed servers and clients and also to work well with mobile computers that operate while detached from the network for long periods. It has not seen wide commercial use, but it continues as a research vehicle.

In the Microsoft family of operating systems, remote filesystem access is provided by the Common Internet File System (CIFS), which runs on top of the Server Message Block (SMB) [SNIA, 2002]. In FreeBSD, support for SMB and CIFS client and server is provided by Samba, which resides in /usr/ports/net/samba. Since this book deals with the kernel, and Samba runs mostly external to the kernel, we will not discuss it further.

The most commercially successful and widely available remote-filesystem protocol is the network filesystem (NFS), originally designed and implemented by Sun Microsystems [Sandberg et al., 1985; Walsh et al., 1985]. There are two important components to the success of NFS. First, Sun placed the protocol specification for NFS in the public domain. Second, Sun sells that implementation to anyone who wants it, for less than the cost of implementing it themselves. Thus, most vendors chose to buy the Sun implementation. They are willing to buy from Sun because they know that they can always legally write their own implementation if the price of the Sun implementation goes up unreasonably. The 4.4BSD implementation was written from the protocol specification rather than being incorporated from Sun because of the developers' desire to be able to redistribute it freely in source form.

The first widely released implementation of NFS was version 2 by Sun in 1984. Although version 3 was expected to be released within a year or two of version 2, it suffered several iterations of hugely complicated proposals before an incremental improvement on version 2 was finally released in 1992. The final release of 4.4BSD included an implementation of NFS that supported both versions 2 and 3. FreeBSD's NFS implementation is a direct descendant of the code released in 4.4BSD. The changes involve bug fixes and performance improvements along with keeping it working with the ever-evolving vnode interface.

Although versions 2 and 3 of NFS were done entirely within Sun, the growing set of companies providing NFS-based products put increasing pressure on Sun to bring others into the design of NFS version 4. After much political maneuvering, Sun agreed to pass the responsibility for defining the specification of NFS version 4 to the Internet Engineering Task Force (IETF). Version 4 greatly expands the functionality of the earlier versions of NFS. It incorporates the mounting validation and locking that used to be done by separate daemons running their own protocols. File access and attribute semantics are expanded to interoperate more easily with other important protocols such as CIFS. At publication time (2004), the NFS version 4 specification is still a draft standard [Shepler et al., 2003]. There are people who are actively working on writing an NFS version 4 implementation for BSD, but it has not yet been tried in FreeBSD. The rest of this chapter examines the NFS version 3 that is currently deployed in FreeBSD.

NFS was designed as a client-server application. Its implementation is divided into a client part that imports filesystems from other machines and a server part that exports local filesystems to other machines. The general model is shown in Figure 9.1. In FreeBSD, the kernel can be configured to support just the client, just the server, or both client and server. Many goals went into the NFS design:

  • The protocol is designed to be stateless. Because there is no state to maintain or recover, NFS can continue to operate even during periods of client or server failures. Thus, it is much more robust than a system that operates with state.

  • NFS is designed to support UNIX filesystem semantics. However, its design also allows it to support the possibly less rich semantics of other filesystem types, such as MS-DOS.

  • The protection and access controls follow the UNIX semantics of having the process present a UID and set of groups that are checked against the file's owner, group, and other access modes. The security check is done by filesystem-dependent code that can do more or fewer checks based on the capabilities of the filesystem that it is supporting. For example, the MS-DOS filesystem cannot implement the full UNIX security validation, and it makes access decisions solely based on the UID.

  • The protocol design is transport independent. Although it was originally built using the UDP datagram protocol in version 2, it was easily moved to the TCP stream protocol in version 3. It has also been ported to run over numerous other non-IP-based protocols.

Figure 9.1. The division of NFS between client and server.

Some of the design decisions limit the set of applications for which NFS is appropriate:

  • The design envisions clients and servers being connected on a locally fast network. The NFS protocol does not work well over slow links. When using UDP as the transport, it does not work well between clients and servers with intervening gateways. It also works poorly for mobile computing that has extended periods of disconnected operation.

  • The caching model assumes that most files will not be shared. Performance suffers when files are heavily shared.

  • The stateless protocol requires some loss of traditional UNIX semantics. Filesystem locking (flock) has to be implemented by a separate stateful daemon. Deferral of the release of space in an unlinked file until the final process has closed the file is approximated with a heuristic that sometimes fails.

Despite these limitations, NFS proliferated because it makes a reasonable tradeoff between semantics and performance; its low cost of adoption has now made it ubiquitous.


9.2. NFS Structure and Operation

NFS operates as a typical client-server application. The server receives remote-procedure-call (RPC) requests from its various clients. An RPC operates much like a local procedure call: The client makes a procedure call and then waits for the result while the procedure executes. For a remote procedure call, the parameters must be marshalled together into a message. Marshalling includes replacing pointers by the data to which they point and converting binary data to the canonical network byte order. The message is then sent to the server, where it is unmarshalled (separated out into its original pieces) and processed as a local filesystem operation. The result must be similarly marshalled and sent back to the client.

The client splits up the result and returns that result to the calling process as though the result were being returned from a local procedure call [Birrell & Nelson, 1984]. The NFS protocol uses Sun's RPC and external data-representation (XDR) protocols [Reid, 1987]. Although the kernel implementation is done by hand to get maximum performance, the user-level daemons described later in this section use Sun's public-domain RPC and XDR libraries.

The NFS protocol can run over any available stream- or datagram-oriented protocol. Common choices are the TCP stream protocol and the UDP datagram protocol. Each NFS RPC message may need to be broken into multiple packets to be sent across the network. A big performance problem for NFS running under UDP on an Ethernet is that the message may be broken into up to six packets; if any of these packets are lost, the entire message is lost and must be resent. When running under TCP on an Ethernet, the message may also be broken into up to six packets; however, individual lost packets, rather than the entire message, can be retransmitted. Section 9.3 discusses performance issues in greater detail.

The set of RPC requests that a client can send to a server is shown in Table 9.1 (on page 390). After the server handles each request, it responds with the appropriate data or with an error code explaining why the request could not be done. As noted in the table, many operations are idempotent. An idempotent operation is one that can be repeated several times without the final result being changed or an error being caused. For example, writing the same data to the same offset in a file is idempotent because it will yield the same result whether it is done once or many times. However, trying to remove the same file more than once is nonidempotent because the file will no longer exist after the first try. Idempotency is an issue when the server is slow or when an RPC acknowledgment is lost and the client retransmits the RPC request. The retransmitted RPC will cause the server to try to do the same operation twice. For a nonidempotent request, such as a request to remove a file, the retransmitted RPC, if undetected by the server recent-request cache [Juszczak, 1989], will cause a "no such file" error to be returned, because the file will have been removed already by the first RPC. The user may be confused by the error because they will have successfully found and removed the file.

Table 9.1. NFS, Version 3, RPC requests.

RPC request

Action

Idempotent

GETATTR

get file attributes

yes

SETATTR

set file attributes

yes

LOOKUP

look up file name

yes

ACCESS[ [image]]

check access permission

yes

READLINK

read from symbolic link

yes

READ

read from file

yes

WRITE

write to file

yes

COMMIT[ [image]]

commit cached data on a server to stable storage

yes

CREATE

create file

no

REMOVE

remove file

no

RENAME

rename file

no

LINK

create link to file

no

SYMLINK

create symbolic link

no

MKNOD[ [image]]

create a special device

no

MKDIR

create directory

no

RMDIR

remove directory

no

READDIR

read from directory

yes

READDIRPLUS[ [image]]

extended read from directory

yes

FSSTAT

get dynamic filesystem attributes

yes

FSINFO[ [image]]

get static filesystem attributes

yes

PATHCONF[ [image]]

retrieve POSIX information

yes

[ [image]] RPC requests added in version 3

Each file on the server can be identified by a unique file handle. A file handle is the token by which clients refer to files on a server. Handles are globally unique and are passed in operations, such as read and write, that reference a file. A file handle is created by the server when a pathname-translation request (lookup) is sent from a client to the server. The server must find the requested file or directory and ensure that the requesting user has access permission. If permission is granted, the server returns a file handle for the requested file to the client. The file handle identifies the file in future access requests by the client. Servers are free to build file handles from whatever information they find convenient. In the FreeBSD NFS implementation, the file handle is built from a filesystem identifier, an inode number, and a generation number. The server creates a unique filesystem identifier for each of its locally mounted filesystems. A generation number is assigned to an inode each time that the latter is allocated to represent a new file. The generation number is selected by using the kernel's random-number generator. The kernel ensures that the same value is never used for two consecutive allocations of the inode.

The purpose of the file handle is to provide the server with enough information to find the file in future requests. The filesystem identifier and inode provide a unique identifier for the inode to be accessed. The generation number verifies that the inode still references the same file that it referenced when the file was first accessed. The generation number detects when a file has been deleted, and a new file is later created using the same inode. Although the new file has the same filesystem identifier and inode number, it is a completely different file from the one that the previous file handle referenced. Since the generation number is included in the file handle, the generation number in a file handle for a previous use of the inode will not match the new generation number in the same inode. When an old-generation file handle is presented to the server by a client, the server refuses to accept it and instead returns the "stale file handle" error message.

The use of the generation number ensures that the file handle is time stable. Distributed systems define a time-stable identifier as one that refers uniquely to some entity both while that entity exists and for a long time after it is deleted. A time-stable identifier allows a system to remember an identity across transient failures and allows the system to detect and report errors for attempts to access deleted entities.

The NFS Protocol

The NFS protocol is stateless. Being stateless means that the server does not need to maintain any information about which clients it is serving or about the files that they currently have open. Every RPC request that is received by the server is completely self-contained. The server does not need any additional information beyond that contained in the RPC to fulfill the request. For example, a read request will include the credential of the user doing the request, the file handle on which the read is to be done, the offset in the file to begin the read, and the number of bytes to be read. This information allows the server to open the file, verifying that the user has permission to read it; seek to the appropriate point; read the desired contents; and close the file. In practice, the server caches recently accessed file data. However, if there is enough activity to push the file out of the cache, the file handle provides the server with enough information to reopen the file.

In addition to reducing the work needed to service incoming requests, the server cache also detects retries of previously serviced requests. Occasionally, a UDP client will send a request that is processed by the server, but the acknowledgment returned by the server to the client is lost. Receiving no answer, the client will timeout and resend the request. The server will use its cache to recognize that the retransmitted request has already been serviced. Thus, the server will not repeat the operation but will just resend the acknowledgment. To detect such retransmissions properly, the server cache needs to be large enough to keep track of at least the most recent few seconds of NFS requests.

The benefit of the stateless protocol is that there is no need to do state recovery after a client or server has crashed and rebooted or after the network has been partitioned and reconnected. Because each RPC is self-contained, the server can simply begin servicing requests as soon as it begins running; it does not need to know which files its clients have open. Indeed, it does not even need to know which clients are currently using it as a server.

There are drawbacks to the stateless protocol. First, the semantics of the local filesystem imply state. When files are unlinked, they continue to be accessible until the last reference to them is closed. Because NFS knows neither which files are open on clients nor when those files are closed, it cannot properly know when to free file space. As a result, it always frees the space at the time of the unlink of the last name to the file. Clients that want to preserve the freeing-on-last-close semantics convert unlinks of open files to renames to obscure names on the server. The names are in the form. .nfsAxxxx4.4, where the xxxx is replaced with the hexadecimal value of the process identifier, and the A is successively incremented until an unused name is found. When the last close is done on the client, the client sends an unlink of the obscure filename to the server. This heuristic works for file access on only a single client; if one client has the file open and another client removes the file, the file will still disappear from the first client at the time of the remove. Other stateful semantics include the advisory locking described in Section 7.5. The locking semantics cannot be handled by the NFS protocol. On most systems, they are handled by a separate lock manager; the FreeBSD version of NFS implements them using the user-level rpc.lockd daemon.

The second drawback of the stateless protocol is related to performance. For version 2 of the NFS protocol, all operations that modify the filesystem must be committed to stable-storage before the RPC can be acknowledged. Most servers do not have battery-backed memory; the stable store requirement means that all written data must be on the disk before they can reply to the RPC. For a growing file, an update may require up to three synchronous disk writes: one for the inode to update its size, one for the indirect block to add a new data pointer, and one for the new data themselves. Each synchronous write takes several milliseconds; this delay severely restricts the write throughput for any given client file.

Version 3 of the NFS protocol eliminates some of the synchronous writes by adding a new asynchronous write RPC request. When such a request is received by the server, it is permitted to acknowledge the RPC without writing the new data to stable storage. Typically, a client will do a series of asynchronous write requests followed by a commit RPC request when it reaches the end of the file or it runs out of buffer space to store the file. The commit RPC request causes the server to write any unwritten parts of the file to stable store before acknowledging the commit RPC. The server benefits by having to write the inode and indirect blocks for the file only once per batch of asynchronous writes, instead of on every write RPC request. The client benefits from having higher throughput for file writes. The client does have the added overhead of having to save copies of all asynchronously written buffers until a commit RPC is done because the server may crash before having written one or more of the asynchronous buffers to stable store. Each time the client does an asynchronous write RPC, the server returns a verification token. When the client sends the commit RPC, the acknowledgment to that RPC also includes a verification token. The verification token is a cookie that the client can use to determine whether the server has rebooted between a call to write data and a subsequent call to commit it. The cookie is guaranteed to be the same throughout a single boot session of the server and to be different each time the server reboots where uncommitted data may be lost. If the verification token changes, the client knows that they must retransmit all asynchronous write RPCs done since the last commit RPC that were verified with the old verification-token value.

The NFS protocol does not specify the granularity of the buffering that should be used when files are written. Most implementations of NFS buffer files in 8-Kbyte blocks. Thus, if an application writes 10 bytes in the middle of a block, the client reads the entire block from the server, modifies the requested 10 bytes, and then writes the entire block back to the server. The FreeBSD implementation also uses 8-Kbyte buffers, but it keeps additional information that describes which bytes in the buffer are modified. If an application writes 10 bytes in the middle of a block, the client reads the entire block from the server, modifies the requested 10 bytes but then writes back only the 10 modified bytes to the server. The block read is necessary to ensure that, if the application later reads back other unmodified parts of the block, it will get valid data. Writing back only the modified data has two benefits:

  1. Fewer data are sent over the network, reducing contention for a scarce resource.

  2. Nonoverlapping modifications to a file are not lost. If two different clients simultaneously modify different parts of the same file block, both modifications will show up in the file, since only the modified parts are sent to the server. When clients send back entire blocks to the server, changes made by the first client will be overwritten by data read before the first modification was made and then will be written back by the second client.

The FreeBSD NFS Implementation

The NFS implementation that appears in FreeBSD was written by Rick Macklem at the University of Guelph, using the specifications of the Version 2 protocol published by Sun Microsystems [Macklem, 1991; Sun Microsystems, 1989]. He later extended it to support the protocol extensions found in the version 3 protocol [Callaghan et al., 1995; Pawlowski et al., 1994]. Table 9.1 highlights the new functionality in the version 3 protocol. The version 3 protocol provides the following:

  • Sixty-four-bit file offsets and sizes

  • An access RPC that provides server permission checking on file open, rather than having the client guess whether the server will allow access

  • An append option on the write RPC

  • A defined way to make special device nodes and fifos

  • Optimization of bulk directory access

  • The ability to batch writes into several asynchronous RPCs followed by a commit RPC to ensure that the data are on stable storage

  • Additional information about the capabilities of the underlying filesystem

In addition to the version 2 and version 3 support, Rick Macklem made several other extensions to the BSD NFS implementation; the extended version became known as the Not Quite NFS (NQNFS) protocol [Macklem, 1994a]. The NQNFS extensions add the following:

  • Extended file attributes to support FreeBSD filesystem functionality more fully

  • A variant of short-term leases with delayed-write client caching that give distributed cache consistency and improved performance [Gray & Cheriton, 1989]

Although the NQNFS extensions were never widely adopted in version 3 implementations, they were instrumental in proving the value of using leases in NFS. The leasing technology was adopted for use in the NFS version 4 protocol, not only for cache consistency and improved performance of the files and directories, but also as a mechanism to bound the recovery time for locks.

The NFS implementation distributed in FreeBSD supports clients and servers running the NFS version 2, NFS version 3, or NQNFS protocol [Macklem, 1994b]. The NQNFS protocol is described in Section 9.3.

The FreeBSD client and server implementations of NFS are kernel resident. NFS interfaces to the network with sockets using the kernel interface available through sosend() and soreceive() (see Chapter 11 for a discussion of the socket interface). There are connection-management routines for support of sockets using connection-oriented protocols; there are timeout and retransmit support for datagram sockets on the client side.

The less time-critical operations, such as mounting and unmounting, as well as determination of which filesystems may be exported and to what set of clients they may be exported are managed by user-level system daemons. For the server side to function, the portmap, mountd, and nfsd daemons must be running. For full NFS functionality, the rpc.lockd and rpc.statd daemons must also be running.

The portmap daemon acts as a registration service for programs that provide RPC-based services. When an RPC daemon is started, it tells the portmap daemon to what port number it is listening and what RPC services it is prepared to serve. When a client wishes to make an RPC call to a given service, it will first contact the portmap daemon on the server machine to determine the port number to which RPC messages should be sent.

The interactions between the client and server daemons when a remote filesystem is mounted are shown in Figure 9.2. The mountd daemon handles two important functions:

  1. On startup and after a hangup signal, mountd reads the /etc/exports file and creates a list of hosts and networks to which each local filesystem may be exported. It passes this list into the kernel using the mount system call; the kernel links the list to the associated local filesystem mount structure so that the list is readily available for consultation when an NFS request is received.

  2. Client mount requests are directed to the mountd daemon. After verifying that the client has permission to mount the requested filesystem, mountd returns a file handle for the requested mount point. This file handle is used by the client for later traversal into the filesystem.

Figure 9.2. Daemon interaction when a remote filesystem is mounted. Step 1: The client's mount process sends a message to the well-known port of the server's portmap daemon, requesting the port address of the server's mountd daemon. Step 2: The server's portmap daemon returns the port address of its server's mountd daemon. Step 3: The client's mount process sends a request to the server's mountd daemon with the pathname of the filesystem that it wants to mount. Step 4: The server's mountd daemon requests a file handle for the desired mount point from its kernel. If the request is successful, the file handle is returned to the client's mount process. Otherwise, the error from the file-handle request is returned. If the request is successful, the client's mount process does a mount system call, passing in the file handle that it received from the server's mountd daemon.

The nfsd master daemon forks off children that enter the kernel using the nfssvc system call. The children normally remain kernel resident, providing a process context for the NFS RPC daemons. Typical systems run four to six nfsd daemons. If nfsd is providing datagram service, it will create a datagram socket when it is started. If nfsd is providing stream service, connected stream sockets will be passed in by the master nfsd daemon in response to connection-oriented connection requests from clients. When a request arrives on a datagram or stream socket, there is a callback from the socket layer that invokes the nfsrv_rcv() routine. The nfsrv_rcv() call takes the message from the socket receive queue and dispatches that message to an available nfsd daemon. The nfsd daemon verifies the sender and then passes the request to the appropriate local filesystem for processing. When the result returns from the filesystem, it is returned to the requesting client. The nfsd daemon is then ready to loop back and to service another request. The maximum degree of concurrency on the server is determined by the number of nfsd daemons that are started.

For connection-oriented transport protocols, such as TCP, there is one connection for each client-to-server mount point. For datagram-oriented protocols, such as UDP, the server creates a fixed number of incoming RPC sockets when it starts its nfsd daemons; clients create one socket for each imported mount point. The socket for a mount point is created by the mount command on the client, which then uses it to communicate with the mountd daemon on the server. Once the client-to-server connection is established, the daemon processes on a connection-oriented protocol may do additional verification, such as Kerberos authentication. Once the connection is created and verified, the socket is passed into the kernel. If the connection breaks while the mount point is still active, the client will attempt a reconnect with a new socket.

The rpc.lockd daemon manages locking requests for remote files. Client locking requests are exported from the kernel through a fifo, /var/run/lock. The rpc.lockd daemon reads the locking request from the fifo and sends them across the network to the rpc.lockd daemon on the server that holds the file. The daemon running on the server opens the file to be locked and uses the locking primitives described in Section 8.5 to acquire the requested lock. Once the lock has been acquired, the server daemon sends a message back to the client daemon. The client daemon writes the lock status into the fifo, which is then read by the kernel and passed up to the user application. The release of the lock is handled similarly. If the rpc.lockd daemon is not run, then lock requests on NFS files will fail with an "operation not supported" error.

The rpc.statd daemon cooperates with rpc.statd daemons on other hosts to provide a status monitoring service. The daemon accepts requests from programs running on the local host (typically rpc.lockd) to monitor the status of specified hosts. If a monitored host crashes and restarts, the daemon on the crashed host will notify the other daemons that it crashed when it is restarted. When notified of a crash, or when a daemon determines that a remote host has crashed because of its lack of response, it will notify the local program(s) that requested the monitoring service. If the rpc.statd daemon is not run, then locks held by clients on a host that crashed may be held indefinitely. By using the rpc.statd service, crashes will be discovered and the locks held by a crashed host will be released.

The client side can operate without any daemons running, but the system administrator can improve performance by running several nfsiod daemons (these daemons provide the same service as the Sun biod daemons). As with the server, for full functionality the client must run the rpc.lockd and rpc.statd daemons.

The purpose of the nfsiod daemons is to do asynchronous read-aheads and write-behinds. They are typically started when the kernel begins running multiuser. They enter the kernel using the nfssvc system call, and they remain kernel resident, providing a process context for the NFS RPC client side. In their absence, each read or write of an NFS file that cannot be serviced from the local client cache must be done in the context of the requesting process. The process sleeps while the RPC is sent to the server, the RPC is handled by the server, and a reply is sent back. No read-aheads are done, and write operations proceed at the disk-write speed of the server. When present, the nfsiod daemons provide a separate context in which to issue RPC requests to a server. When a file is written, the data are copied into the buffer cache on the client. The buffer is then passed to a waiting nfsiod that does the RPC to the server and awaits the reply. When the reply arrives, nfsiod updates the local buffer to mark that buffer as written. Meanwhile, the process that did the write can continue running. The Sun Microsystems reference implementation of the NFS protocol flushes all the blocks of a file to the server when that file is closed. If all the dirty blocks have been written to the server when a process closes a file that it has been writing, it will not have to wait for them to be flushed. The NQNFS protocol does not flush all the blocks of a file to the server when that file is closed.

When reading a file, the client first hands a read-ahead request to the nfsiod that does the RPC to the server. It then looks up the buffer that it has been requested to read. If the sought-after buffer is already in the cache because of a previous read-ahead request, then it can proceed without waiting. Otherwise, it must do an RPC to the server and wait for the reply. The interactions between the client and server daemons when I/O is done are shown in Figure 9.3.

Figure 9.3. Daemon interaction when I/O is done. Step 1: The client's process does a write system call. Step 2: The data to be written are copied into a kernel buffer on the client, and the write system call returns. Step 3: An nfsiod daemon awakens inside the client's kernel, picks up the dirty buffer, and sends the buffer to the server. Step 4: The incoming write request is delivered to the next available nfsd daemon running inside the kernel on the server. The server's nfsd daemon writes the data to the appropriate local disk and waits for the disk I/O to complete. Step 5: After the I/O has completed, the server's nfsd daemon sends back an acknowledgment of the I/O to the waiting nfsiod daemon on the client. On receipt of the acknowledgment, the client's nfsiod daemon marks the buffer as clean.

Client-Server Interactions

A local filesystem is unaffected by network service disruptions. It is always available to the users on the machine unless there is a catastrophic event, such as a disk or power failure. Since the entire machine hangs or crashes, the kernel does not need to concern itself with how to handle the processes that were accessing the filesystem. By contrast, the client end of a network filesystem must have ways to handle processes that are accessing remote files when the client is still running but the server becomes unreachable or crashes. Each NFS mount point is provided with three alternatives for dealing with server unavailability:

  1. The default is a hard mount that will continue to try to contact the server indefinitely to complete the filesystem access. This type of mount is appropriate when processes on the client that access files in the filesystem do not tolerate I/O system calls that return transient errors. A hard mount is used for processes for which access to the filesystem is critical for normal system operation. It is also useful if the client has a long-running program that simply wants to wait for the server to resume operation (e.g., after the server is taken down for maintenance).

  2. The other extreme is a soft mount that retries an RPC a specified number of times, and then the corresponding system call returns with a transient error. For a connection-oriented protocol, the actual RPC request is not retransmitted; instead, NFS depends on the protocol retransmission to do the retries. If a response is not returned within the specified time, the corresponding system call returns with a transient error. The problem with this type of mount is that most applications do not expect a transient error return from I/O system calls (since they never occur on a local filesystem). Often, they will mistakenly interpret the transient error as a permanent error and will exit prematurely. An additional problem is deciding how long to set the timeout period. If it is set too low, error returns will start occurring whenever the NFS server is slow because of heavy load. Alternately, a large retry limit can result in a process hung for a long time because of a crashed server or network partitioning.

  3. Most system administrators take a middle ground by using an interruptible mount that will wait forever like a hard mount but checks to see whether a termination signal is pending for any process that is waiting for a server response. If a signal (such as an interrupt) is sent to a process waiting for an NFS server, the corresponding I/O system call returns with a transient error. Normally, the process is terminated by the signal. If the process chooses to catch the signal, then it can decide how to handle the transient failure. This mount option allows interactive programs to be aborted when a server fails, while allowing long-running processes to await the server's return.

The original NFS implementation had only the first two options. Since neither of these two options was ideal for interactive use of the filesystem, the third option was developed as a compromise solution.

RPC Transport Issues

The NFS version 2 protocol runs over UDP/IP transport by sending each request-reply message in a single UDP datagram. Since UDP does not guarantee datagram delivery, a timer is started, and if a timeout occurs before the corresponding RPC reply is received, the RPC request is retransmitted. At best, an extraneous RPC request retransmit increases the load on the server and can result in damaged files on the server or spurious errors being returned to the client when nonidempotent RPCs are redone. A recent-request cache normally is used on the server to minimize the negative effect of redoing a duplicate RPC request [Juszczak, 1989]. The recent-request cache keeps copies of all the nonidempotent RPC requests that the server has received over the past few minutes (usually up to 15 minutes) along with the response that it sent to that request. Each incoming request is checked against those in the cache. If a request matches one to which the server previously responded, the previously calculated result is returned instead of rerunning the request. Thus, the client gets the same result that it would have received earlier rather than an incorrect replay. The recent-request cache prevents most of the nonidempotent-related errors. However, a network partition that exceeds the timeout value for entries in the cache can still cause nonidempotent errors to be visible.

The amount of time that the client waits before resending an RPC request is called the round-trip timeout (RTT). Figuring out an appropriate value for the RTT is difficult. The RTT value is for the entire RPC operation, including transmitting the RPC message to the server, queueing at the server for an nfsd, doing any required I/O operations, and sending the RPC reply message back to the client. It can be highly variable for even a moderately loaded NFS server. As a result, the RTT interval must be a conservative (large) estimate to avoid extraneous RPC request retransmits. Adjusting the RTT interval dynamically and applying a congestion window on outstanding requests has been shown to be of some help with the retransmission problem [Nowicki, 1989].

When sending an NFS default-sized 8-Kbyte read-write data request, the read-write reply-request will be an 8+-Kbyte UDP datagram. On an Ethernet with a maximum-transmission unit (MTU) of 1500 bytes, an 8+-Kbyte datagram must be broken into at least six fragments at the IP layer for transmission. For IP fragments to be reassembled successfully into the IP datagram at the receive end, all fragments must be received at the destination. If even one fragment is lost or damaged in transit, the entire RPC message must be retransmitted and the entire RPC redone. This problem can be exaggerated if the server is multiple hops away from the client through routers or slow links. It can also be nearly fatal if the network interface on the client or server cannot handle the reception of back-to-back network packets [Kent & Mogul, 1987].

An alternative to all this madness is to run NFS over TCP transport instead of over UDP. Since TCP provides reliable delivery with congestion control, it avoids the problems associated with UDP. Because the retransmissions are done at the TCP level, instead of at the RPC level, the only time that a duplicate RPC will be sent to the server is when the server crashes or there is an extended network partition that causes the TCP connection to break after an RPC has been received but not acknowledged to the client. Here, the client will resend the RPC after the server reboots, because it does not know that the RPC has been received.

The use of TCP also permits the use of read and write data sizes greater than the 8-Kbyte limit for UDP transport. Using large data sizes allows TCP to use the full duplex bandwidth of the network effectively before being forced to stop and wait for RPC response from the server. NFS over TCP usually delivers comparable to significantly better performance than NFS over UDP, unless the client or server processor is CPU limited. Here the extra CPU overhead of using TCP transport becomes significant.

The main problem with using TCP transport with version 2 of NFS is that it is supported between only BSD and a few other vendor's clients and servers. However, the clear superiority demonstrated by the version 2 BSD TCP implementation of NFS convinced the group at Sun Microsystems implementing NFS version 3 to make TCP the default transport. Thus, a version 3 Sun client will first try to connect using TCP; only if the server refuses will it fall back to using UDP. The version 4 protocol also is defined to use TCP.

Security Issues

NFS versions 2 and 3 are not secure because the protocol was not designed with security in mind. Despite several attempts to fix security problems in these versions, NFS security is still limited. In particular, the security work only addresses authentication; file data are sent around the net in clear text. Even if someone is unable to get your server to send them a sensitive file, they can just wait until a legitimate user accesses it, and then can pick it up as it goes by on the net. Much of the work that went into version 4 addressed both authentication and data security. Once version 4 moves into general use, NFS filesystems will be able to be run reasonably securely.

NFS export control is at the granularity of local filesystems. Associated with each local filesystem mount point is a list of the hosts to which that filesystem may be exported. A local filesystem may be exported to a specific host, to all hosts that match a subnet mask, or to all other hosts (the world). For each host or group of hosts, the filesystem can be exported read-only or read-write. In addition, a server may specify a set of subdirectories within the filesystem that may be mounted. However, this list of mount points is enforced by only the mountd daemon. If a malicious client wishes to do so, it can access any part of a filesystem that is exported to it.

The final determination of exportability is made by the list maintained in the kernel. So even if a rogue client manages to snoop the net and to steal a file handle for the mount point of a valid client, the kernel will refuse to accept the file handle unless the client presenting that handle is on the kernel's export list. When NFS is running with TCP, the check is done once when the connection is established. When NFS is running with UDP, the check must be done for every RPC request.

The NFS server also permits limited remapping of user credentials. Typically, the credential for the superuser is not trusted and is remapped to the low-privilege user "nobody." The credentials of all other users can be accepted as given or also mapped to a default user (typically "nobody"). Use of the client UID and GID list unchanged on the server implies that the UID and GID space are common between the client and server (i.e., UID N on the client must refer to the same user on the server). The system administrator can support more complex UID and GID mappings by using the umapfs filesystem described in Section 6.7.

The system administrator can increase security by using Kerberos credentials instead of accepting arbitrary user credentials sent without encryption by clients of unknown trustworthiness [Steiner et al., 1988]. When a new user on a client wants to begin accessing files in an NFS filesystem that is exported using Kerberos, the client must provide a Kerberos ticket to authenticate the user on the server. If successful, the system looks up the Kerberos principal in the server's password and group databases to get a set of credentials and passes in to the server nfsd a local translation of the client UID to these credentials. The nfsd daemons run entirely within the kernel except when a Kerberos ticket is received. To avoid putting all the Kerberos authentication into the kernel, the nfsd returns from the kernel temporarily to verify the ticket using the Kerberos libraries and then returns to the kernel with the results.

The NFS implementation with Kerberos uses encrypted timestamps to avert replay attempts. Each RPC request includes a timestamp that is encrypted by the client and decrypted by the server using a session key that has been exchanged as part of the initial Kerberos authentication. Each timestamp can be used only once, and it must be within a few minutes of the current time recorded by the server. This implementation requires that the client and server clocks be kept within a few minutes of synchronization (this requirement is already imposed to run Kerberos). It also requires that the server keep copies of all timestamps that it has received that are within the time range that it will accept so that it can verify that a time-stamp is not being reused. Alternatively, the server can require that timestamps from each of its clients be monotonically increasing. However, this algorithm will cause RPC requests that arrive out of order to be rejected. The mechanism of using Kerberos for authentication of NFS requests is not well defined, and the FreeBSD implementation has not been tested for interoperability with other vendors. Thus, Kerberos can be used only between FreeBSD clients and servers.


9.3. Techniques for Improving Performance

Remote filesystems provide a challenging performance problem: Providing both a coherent networkwide view of the data and delivering that data quickly are often conflicting goals. The server can maintain coherency easily by keeping a single repository for the data and sending them out to each client when the clients need them; this approach tends to be slow, because every data access requires the client to wait for an RPC round-trip time. The delay is further aggravated by the huge load that it puts on a server that must service every I/O request from its clients. To increase performance and to reduce server load, remote filesystem protocols attempt to cache frequently used data on the clients themselves. If the cache is designed properly, the client will be able to satisfy many of the client's I/O requests directly from the cache. Doing such accesses is faster than communicating with the server, reducing latency on the client and load on the server and network. The hard part of client caching is keeping the caches coherent—that is, ensuring that each client quickly replaces any cached data that are modified by writes done on other clients. If a first client writes a file that is later read by a second client, the second client wants to see the data written by the first client, rather than the stale data that were in the file previously. There are two main ways that the stale data may be read accidentally:

  1. If the second client has stale data sitting in its cache, the client may use those data because it does not know that newer data are available.

  2. The first client may have new data sitting in its cache but may not yet have written those data back to the server. Here, even if the second client asks the server for up-to-date data, the server may return the stale data because it does not know that one of its clients has a newer version of the file in that client's cache.

The second of these problems is related to the way that client writing is done. Synchronous writing requires that all writes be pushed through to the server during the write system call. This approach is the most consistent because the server always has the most recently written data. It also permits any write errors, such as "filesystem out of space," to be propagated back to the client process via the write system-call return. With an NFS filesystem using synchronous writing, error returns most closely parallel those from a local filesystem. Unfortunately, this approach restricts the client to only one write per RPC round-trip time.

An alternative to synchronous writing is delayed writing, where the write system call returns as soon as the data are cached on the client; the data are written to the server sometime later. This approach permits client writing to occur at the rate of local storage access up to the size of the local cache. Also, for cases where file truncation or deletion occurs shortly after writing, the write to the server may be avoided entirely because the data have already been deleted. Avoiding the data push saves the client time and reduces load on the server.

There are some drawbacks to delayed writing. To provide full consistency, the server must notify the client when another client wants to read or write the file so that the delayed writes can be written back to the server. There are also problems with the propagation of errors back to the client process that issued the write system call. For example, a semantic change is introduced by delayed-write caching when the file server is full. Here, delayed-write RPC requests can fail with an "out of space" error. If the data are sent back to the server when the file is closed, the error can be detected only if the application checks the return value from the close system call. For delayed writes, written data may not be sent back to the server until after the process that did the write has exited—long after it can be notified of any errors. The only solution is to modify programs writing an important file to do an fsync system call and to check for an error return from that call instead of depending on getting errors from write or close. Finally, there is a risk of the loss of recently written data if the client crashes before the data are written back to the server.

A compromise between synchronous writing and delayed writing is asynchronous writing. The write to the server is started during the write system call, but the write system call returns before the write completes. This approach minimizes the risk of data loss because of a client crash but negates the possibility of reducing server write load by discarding writes when a file is truncated or deleted.

The simplest mechanism for maintaining full cache consistency is the one used by Sprite that disables all client caching of the file whenever concurrent write sharing might occur [Nelson et al., 1988]. Since NFS has no way of knowing when write sharing might occur, it tries to bound the period of inconsistency by writing the data back when a file is closed. Files that are open for long periods are written back when their oldest dirty data becomes 30 seconds old. Thus, the NFS implementation does a mix of asynchronous and delayed writing, but it always pushes all writes to the server on close. Pushing the delayed writes on close negates much of the performance advantage of delayed writing because the delays that were avoided in the write system calls are observed in the close system call. With this approach, the server is always aware of all changes made by its clients with a maximum delay of 30 seconds and usually sooner, because most files are open only briefly for writing.

The server maintains read consistency by always having a client verify the contents of its cache before using that cache. When a client reads data, it first checks for the data in its cache. Each cache entry is stamped with an attribute that shows the most recent time that the server says that the data were modified. If the data are found in the cache, the client sends a timestamp RPC request to its server to find out when the data were last modified. If the modification time returned by the server matches that associated with the cache, the client uses the data in its cache; otherwise, it arranges to replace the data in its cache with the new data.

The problem with checking with the server on every cache access is that the client still experiences an RPC round-trip delay for each file access, and the server is still inundated with RPC requests, although they are considerably quicker to handle than are full I/O operations. To reduce this client latency and server load, most NFS implementations track how recently the server has been asked about each cache block. The client then uses a tunable parameter that is typically set at a few seconds to delay asking the server about a cache block. If an I/O request finds a cache block and the server has been asked about the validity of that block within the delay period, the client does not ask the server again, but just uses the block. Because certain blocks are used many times in succession, the server will be asked about them only once, rather than on every access. For example, the directory block for the /usr/include directory will be accessed once for each #include in a source file that is being compiled. The drawback to this approach is that changes made by other clients may not be noticed for up to the delay number of seconds.

A more consistent approach used by some network filesystems is to use a callback scheme where the server keeps track of all the files that each of its clients has cached. When a cached file is modified, the server notifies the clients holding that file so that they can purge it from their cache. This algorithm dramatically reduces the number of queries from the client to the server, with the effect of decreasing client I/O latency and server load [Howard et al., 1988]. The drawback is that this approach introduces state into the server because the server must remember the clients that it is serving and the set of files that they have cached. If the server crashes, it must rebuild this state before it can begin running again. Rebuilding the server state is a significant problem when everything is running properly; it gets even more complicated and time-consuming when it is aggravated by network partitions that prevent the server from communicating with some of its clients [Mogul, 1993].

The FreeBSD NFS implementation uses asynchronous writes while a file is open but synchronously waits for all data to be written when the file is closed. This approach gains the speed benefit of writing asynchronously, yet ensures that any delayed errors will be reported no later than the point at which the file is closed. The implementation will query the server about the attributes of a file at most once every 3 seconds. This 3-second period reduces network traffic for files accessed frequently, yet ensures that any changes to a file are detected with no more than a 3-second delay. Although these heuristics provide tolerable semantics, they are noticeably imperfect. More consistent semantics at lower cost are available with the NQNFS lease protocol described in the next section.

Leases

The NQNFS protocol is designed to maintain full cache consistency between clients in a crash-tolerant manner. It is an adaptation of the NFS protocol such that the server supports both NFS and NQNFS clients while maintaining full consistency between the server and NQNFS clients. The protocol maintains cache consistency by using short-term leases instead of hard-state information about open files [Gray & Cheriton, 1989]. A lease is a ticket permitting an activity that is valid until some expiration time. As long as a client holds a valid lease, it knows that the server will give it a callback if the file status changes. Once the lease has expired, the client must contact the server if it wants to use the cached data.

Leases are issued using time intervals rather than absolute times to avoid the requirement of time-of-day clock synchronization. There are three important time constants known to the server. The maximum_lease_term sets an upper bound on lease duration—typically, 30 seconds to 1 minute. The clock_skew is added to all lease terms on the server to correct for differing clock speeds between the client and server. The write_slack is the number of seconds that the server is willing to wait for a client with an expired write-caching lease to push dirty writes.

Contacting the server after the lease has expired is similar to the NFS technique for reducing server load by checking the validity of data only every few seconds. The main difference is that the server tracks its clients' cached files, so there are never periods of time when the client is using stale data. Thus, the time used for leases can be considerably longer than the few seconds that clients are willing to tolerate possibly stale data. The effect of this longer lease time is to reduce the number of server calls almost to the level found in a full callback implementation, such as the Andrew Filesystem [Howard et al., 1988]. Unlike the callback mechanism, state recovery with leases is trivial. The server needs only to wait for the lease's expiration time to pass, and then to resume operation. Once all the leases have expired, the clients will always communicate with the server before using any of their cached data. The lease expiration time is usually shorter than the time it takes most servers to reboot, so the server can effectively resume operation as soon as it is running. If the machine does manage to reboot more quickly than the lease expiration time, then it must wait until all leases have expired before resuming operation.

An additional benefit of using leases rather than hard state information is that leases use much less server memory. If each piece of state requires 64 bytes, a large server with thousands of clients and a peak throughput of 10,000 RPC requests per second will typically only use about 1 Mbyte of memory for leases, with a worst case of about 15 Mbyte. Even if a server has exhausted lease storage, it can simply wait a few seconds for a lease to expire and free up a record. By contrast, a server with hard state must store records for all files currently open by all clients. The memory requirements are 30 to 120 Mbyte of memory per 1000 clients served.

Whenever a client wishes to cache data for a file, it must hold a valid lease. There are three types of leases: noncaching, read caching, and write caching. A noncaching lease requires that all file operations be done synchronously with the server. A read-caching lease allows for client data caching, but no file modifications may be done. A write-caching lease allows for client caching of writes for the period of the lease. If a client has cached write data that are not yet written to the server when a write-cache lease has almost expired, it will attempt to extend the lease. If the extension fails, the client is required to push the written data.

If all the clients of a file are reading it, they will all be granted a read-caching lease. A read-caching lease allows one or more clients to cache data, but they may not make any modifications to the data. Figure 9.4 shows a typical read-caching scenario. The vertical solid black lines depict the lease records. Note that the time lines are not drawn to scale, since a client-server interaction will normally take less than 100 milliseconds, whereas the normal lease duration is 30 seconds. Every lease includes the time that the file was last modified on the server. The client can use this timestamp to ensure that its cached data are still current. Initially, client A gets a read-caching lease for the file. Later, client A renews that lease and uses it to verify that the data in its cache are still valid. Concurrently, client B is able to obtain a read-caching lease for the same file.

Figure 9.4. Read-caching leases. Solid vertical lines represent valid leases.

[View full size image]

If a single client wants to write a file and there are no readers of that file, the client will be issued a write-caching lease. A write-caching lease permits delayed write caching but requires that all data be pushed to the server when the lease expires or is terminated by an eviction notice. When a write-caching lease has almost expired, the client will attempt to extend the lease if the file is still open, but it is required to push the delayed writes to the server if renewal fails (see Figure 9.5). The writes may not arrive at the server until after the write lease has expired on the client. A consistency problem is avoided because the server keeps its write lease valid for write_slack seconds longer than the time given in the lease issued to the client. In addition, writes to the file by the lease-holding client cause the lease expiration time to be extended to at least write_slack seconds. This write_slack period is conservatively estimated as the extra time that the client will need to write back any written data that it has cached. If the value selected for write_slack is too short, a write RPC may arrive after the write lease has expired on the server. Although this write RPC will result in another client seeing an inconsistency, that inconsistency is no more problematic than the semantics that NFS normally provides.

Figure 9.5. Write-caching lease. Solid vertical lines represent valid leases.

The server is responsible for maintaining consistency among the NQNFS clients by disabling client caching whenever a server file operation would cause inconsistencies. The possibility of inconsistencies occurs whenever a client has a write-caching lease and any other client or a local operation on the server tries to access the file, or when a modify operation is attempted on a file being read cached by clients. If one of these conditions occurs, then all clients will be issued noncaching leases. With a noncaching lease, all reads and writes will be done through the server, so clients will always get the most recent data. Figure 9.6 shows how read and write leases are replaced by a noncaching lease when there is the potential for write sharing. Initially, the file is read by client A. Later, it is written by client B. While client B is still writing, client A issues another read request. Here, the server sends an "eviction notice" message to client B and then waits for lease termination. Client B writes back its dirty data and then sends a "vacated" message. Finally, the server issues noncaching leases to both clients. In general, lease termination occurs when a "vacated" message has been received from all the clients that have signed the lease or when the lease has expired. The server does not wait for a reply for the message pair "eviction notice" and "vacated," as it does for all other RPC messages. They are sent asynchronously to avoid the server waiting indefinitely for a reply from a dead client.

Figure 9.6. Write-sharing leases. Solid vertical lines represent valid leases.

[View full size image]

A client gets leases either by doing a specific lease RPC or by including a lease request with another RPC. Most NQNFS RPC requests allow a lease request to be added to them. Combining lease requests with other RPC requests minimizes the amount of extra network traffic. A typical combination can be done when a file is opened. The client must do an RPC to get the handle for the file to be opened. It can combine the lease request, because it knows at the time of the open whether it will need a read or a write lease. All leases are at the granularity of a file, because all NFS RPC requests operate on individual files, and NFS has no intrinsic notion of a file hierarchy. Directories, symbolic links, and file attributes may be read cached but are not write cached. The exception is the file-size attribute that is updated during cached writing on the client to reflect a growing file. Leases have the advantage that they are typically required only at times when other I/O operations occur. Thus, lease requests can almost always be piggybacked on other RPC requests, avoiding some of the overhead associated with the explicit open and close RPC required by a long-term callback implementation.

The server handles operations from local processes and from remote clients that are not using the NQNFS protocol by issuing short-term leases for the duration of each file operation or RPC. For example, a request to create a new file will get a short-term write lease on the directory in which the file is being created. Before that write lease is issued, the server will vacate the read leases of all the NQNFS clients that have cached data for that directory. Because the server gets leases for all non-NQNFS activity, consistency is maintained between the server and NQNFS clients, even when local or NFS clients are modifying the filesystem. The NFS clients will continue to be no more or less consistent with the server than they were without leases.

Crash Recovery

The server must maintain the state of all the current leases held by its clients. The benefit of using short-term leases is that, maximum_lease_term seconds after the server stops issuing leases, it knows that there are no current leases left. As such, server crash recovery does not require any state recovery. After rebooting, the server simply refuses to service any RPC requests except for writes (predominantly from clients that previously held write leases) until write_slack seconds after the final lease would have expired. For machines that cannot calculate the time that they crashed, the final-lease expiration time can be estimated safely as

boot_time + maximum_lease_term + write_slack + clock_skew

Here, boot time is the time that the kernel began running after the kernel was booted. With a maximum_lease_term 30 to 60 seconds, and clock_skew and write_slack at most a few seconds, this delay amounts to about 1 minute, which for most systems is taken up with the server rebooting process. When this time has passed, the server will have no outstanding leases. The clients will have had at least write_slack seconds to get written data to the server, so the server should be up-to-date. After this, the server resumes normal operation.

There is another failure condition that can occur when the server is congested. In the worst-case scenario, the client pushes dirty writes to the server but a large request queue on the server delays these writes for more than write_slack seconds. In an effort to minimize the effect of these recovery storms, the server replies "try again later" to the RPC requests that it is not yet ready to service [Baker & Ousterhout, 1991]. The server takes two steps to ensure that all clients have been able to write back their written data. First, a write-caching lease is terminated on the server only when there are have been no writes to the file during the previous write_slack seconds. Second, the server will not accept any requests other than writes until it has not been overloaded during the previous write_slack seconds. A server is considered overloaded when there are pending RPC requests and all its nfsd processes are busy.

Another problem that is solved by short-term leases is how to handle a crashed or partitioned client that holds a lease that the server wishes to vacate. The server detects this problem when it needs to vacate a lease so that it can issue a lease to a second client, and the first client holding the lease fails to respond to the vacate request. Here, the server can simply wait for the first client's lease to expire before issuing the new one to the second client. When the first client reboots or gets reconnected to the server, it simply reacquires any leases it now needs. If a client-to-server network connection is severed just before a write-caching lease expires, the client cannot push the dirty writes to the server. Other clients that can contact the server will continue to be able to access the file and will see the old data. Since the write-caching lease has expired on the client, the client will synchronize with the server as soon as the network connection has been reestablished. This delay can be avoided with a write-through policy.

A detailed comparison of the effects of leases on performance is given in Macklem [1994a]. Briefly, leases are most helpful when a server or network is loaded heavily. Here, leases allow up to 30 to 50 percent more clients to use a network and server before beginning to experience a level of congestion equal to what they would on a network and server that were not using leases. In addition, leases provide better consistency and lower latency for clients, independent of the load.

Although the NQNFS protocol never got adopted outside of the BSD family of operating systems, it did provide a proof of concept that was used to validate the use of leases in the NFS version 4 protocol. In the NFS version 4 protocol, leases are used for both cache consistency and the management of lock state. At the time of publication, there is active work on an implementation of NFS version 4 for BSD that will hopefully be ready for production use in FreeBSD within the next year or two.


Exercises

9.1 Describe the functions done by an NFS client.

9.2 Describe the functions done by an NFS server.

9.3 Describe three benefits that NFS derives from being stateless.

9.4 Give two reasons why TCP is a better protocol to use than is UDP for handling the NFS RPC protocol.

9.5 Describe the contents of a file handle in FreeBSD. How is a file handle used?

9.6 When is a new generation number assigned to a file? What purpose does the generation number serve?

9.7 Describe the three ways that an NFS client can handle filesystem-access attempts when its server crashes or otherwise becomes unreachable.

9.8 Give two reasons why leases are given a limited lifetime.

9.9 What is a callback? When is it used?

9.10 A server may issue three types of leases: noncaching, read caching, and write caching. Describe what a client can do with each of these leases.

9.11 Describe how an NQNFS server recovers after a crash.

*9.12 Suppose that there is a client that supports both versions 2 and 3 of the NFS protocol running on both the TCP and UDP protocols, but a server that supports only version 2 of NFS running on UDP. Show the protocol negotiation between the client and server, assuming that the client prefers to run using version 3 of NFS using TCP.

**9.13 Assume that leases have an unlimited lifetime. Design a system for recovering the lease state after a client or server crash.


References

Baker & Ousterhout, 1991. M. Baker & J. Ousterhout, "Availability in the Sprite Distributed File System," ACM Operating System Review, vol. 25, no. 2, pp. 95-98, April 1991.

Birrell & Nelson, 1984. A. D. Birrell & B. J. Nelson, "Implementing Remote Procedure Calls," ACM Transactions on Computer Systems, vol. 2, no. 1, pp. 39-59, Association for Computing Machinery, February 1984.

Callaghan et al., 1995. B. Callaghan, B. Pawlowski, & P. Staubach, "NFS: Network File System Version 3 Protocol Specification," RFC 1813, available from http://www.faqs.org/rfcs/rfc1813.html, June 1995.

Gray & Cheriton, 1989. C. Gray & D. Cheriton, "Leases: An Efficient Fault-Tolerant Mechanism for Distributed File Cache Consistency," Proceedings of the Twelfth Symposium on Operating Systems Principles, pp. 202-210, December 1989.

Howard, 1988. J. Howard, "An Overview of the Andrew File System," USENIX Association Conference Proceedings, pp. 23-26, January 1988.

Howard et al., 1988. J. Howard, M. Kazar, S. Menees, D. Nichols, M. Satyanarayanan, R. Sidebotham, & M. West, "Scale and Performance in a Distributed File System," ACM Transactions on Computer Systems, vol. 6, no. 1, pp. 51-81, Association for Computing Machinery, February 1988.

Juszczak, 1989. C. Juszczak, "Improving the Performance and Correctness of an NFS Server," USENIX Association Conference Proceedings, pp. 53-63, January 1989.

Kent & Mogul, 1987. C. Kent & J. Mogul, "Fragmentation Considered Harmful," Research Report 87/3, Digital Equipment Corporation Western Research Laboratory, Palo Alto, CA, December 1987.

Macklem, 1991. R. Macklem, "Lessons Learned Tuning the 4.3BSD-Reno Implementation of the NFS Protocol," USENIX Association Conference Proceedings, pp. 53-64, January 1991.

Macklem, 1994a. R. Macklem, "Not Quite NFS, Soft Cache Consistency for NFS," USENIX Association Conference Proceedings, pp. 261-278, January 1994.

Macklem, 1994b. R. Macklem, "The 4.4BSD NFS Implementation," in 4.4BSD System Manager's Manual, pp. 6:1-14, O'Reilly & Associates, Inc., Sebastopol, CA, 1994.

Mogul, 1993. J. Mogul, "Recovery in Spritely NFS," Research Report 93/2, Digital Equipment Corporation Western Research Laboratory, Palo Alto, CA, June 1993.

Nelson et al., 1988. M. Nelson, B. Welch, & J. Ousterhout, "Caching in the Sprite Network File System," ACM Transactions on Computer Systems, vol. 6, no. 1, pp. 134-154, Association for Computing Machinery, February 1988.

Nowicki, 1989. B. Nowicki, "Transport Issues in the Network File System," Computer Communications Review, vol. 19, no. 2, pp. 16-20, April 1989.

Pawlowski et al., 1994. B. Pawlowski, C. Juszczak, P. Staubach, C. Smith, D. Lebel, & D. Hitz, "NFS Version 3: Design and Implementation," USENIX Association Conference Proceedings, pp. 137-151, June 1994.

Reid, 1987. Irving Reid, "RPCC: A Stub Compiler for Sun RPC," USENIX Association Conference Proceedings, pp. 357-366, June 1987.

Rifkin et al., 1986. A. Rifkin, M. Forbes, R. Hamilton, M. Sabrio, S. Shah, & K. Yueh, "RFS Architectural Overview," USENIX Association Conference Proceedings, pp. 248-259, June 1986.

Sandberg et al., 1985. R. Sandberg, D. Goldberg, S. Kleiman, D. Walsh, & B. Lyon, "Design and Implementation of the Sun Network Filesystem," USENIX Association Conference Proceedings, pp. 119-130, June 1985.

Shepler et al., 2003. S. Shepler, B. Callaghan, D. Robinson, R. Thurlow, C. Beame, M. Eisler, & D. Noveck, "Network File System (NFS) version 4 Protocol," RFC 3530, available from http://www.faqs.org/rfcs/rfc3530.html, April 2003.

SNIA, 2002. Storage Networking Industry Association SNIA, "Common Internet File System (CIFS) Technical Reference," www.snia.org/tech_activities/CIFS/CIFS-TR-1p00_FINAL.pdf, March 2002.

Steiner et al., 1988. J. Steiner, C. Neuman, & J. Schiller, "Kerberos: An Authentication Service for Open Network Systems," USENIX Association Conference Proceedings, pp. 191-202, February 1988.

Sun Microsystems, 1989. Sun Microsystems, "NFS: Network File System Protocol Specification," RFC 1094, available from http://www.faqs.org/rfcs/rfcl094.html, March 1989.

Walsh et al., 1985. D. Walsh, B. Lyon, G. Sager, J. Chang, D. Goldberg, S. Kleiman, T. Lyon, R. Sandberg, & P. Weiss, "Overview of the Sun Network File System," USENIX Association Conference Proceedings, pp. 117-124, January 1985.


Chapter 10. Terminal Handling

Historically, users interacted with the system using terminals connected to the computer through hardwired RS-232 serial lines. Although most computers still have one or two hardwired serial lines, their use is mostly relegated to running the system console, slow speed modems, serially connected printers, and similar devices. Although hardwired terminals are largely a footnote in history, the character processing done for keyboard I/O is still important.

The most common type of user session in FreeBSD uses a pseudo-terminal, or pty. The pseudo-terminal driver provides support for a device pair, termed the master and slave devices. The slave device provides to a process an interface identical to the one described for terminals in this chapter. However, instead of being associated with a hardware device to read and write characters, the slave device has another process manipulating it through the master half of the pseudo-terminal. That is, anything written on the master device is provided to the slave device as input, and anything written on the slave device is presented to the master device as input. The driver for the master device emulates all specific hardware support details described in the rest of this chapter.

Pseudo-terminals are used by the terminal emulator, xterm, as well as the common remote-login programs such as ssh and telnet. In a typical use, xterm opens the master side of a pseudo-terminal and directs the keystrokes from the window manager to its input while taking its output and drawing the characters in its window. It forks a process that opens the slave side of the pseudo-terminal and then runs the user's preferred shell with the slave set up as the standard input, output, and error. As each keystroke is typed by the user, it is written into the master side of the pseudo-terminal, where it is processed by the line discipline and eventually emerges as input to the user's shell. Output from the shell is written into the slave side of the pseudo-terminal, where it is processed by the line discipline and eventually emerges from the master side and is displayed in the xterm window.

Like the terminal devices that preceded them, the pseudo-terminal driver commonly processes data one character at a time using the character-device interface described in Section 6.2. As each character is typed at the keyboard or arrives from a user over the network, it is presented as input to the master side of the pseudo-terminal. The input of characters is independent of process requests to read user input from the slave side of the pseudo-terminal. Characters are processed when they are received and then are stored until a process requests them, thus allowing type-ahead. When a pseudo-terminal supports a user interaction with the system, terminal input represents the keystrokes of the user, and terminal output is displayed on the user's screen. We deal mostly with this type of pseudo-terminal usage in this chapter. When we use the term terminal, we are describing a concept that applies to both pseudo-terminals and hardwired terminal devices.

Asynchronous serial lines also connect modems for computer-to-computer communications or serial-interface printers. When serial interfaces are used for these purposes, they generally use a subset of the system's terminal-handling capability. Sometimes, they use special processing modules for higher efficiency. We discuss alternate terminal modules in Section 10.9.


10.1. Terminal-Processing Modes

FreeBSD supports several modes of terminal processing. Much of the time, keyboard input is in canonical mode (also commonly referred to as cooked mode or line mode), in which input characters are echoed by the operating system as they are typed by the user but are buffered internally until a newline character is received. Only after the receipt of a newline character is the entire line made available to the shell or other process reading from the keyboard. If the process attempts to read from the keyboard before a complete line is ready, the process will sleep until a newline character is received, regardless of a partial line already having been received. The common case where a carriage return behaves like a newline character and causes the line to be made available to the waiting process is implemented by the operating system and is configurable by the user or process. In canonical mode, the user may correct typing errors, deleting the most recently typed character with the erase character, deleting the most recent word with the word-erase character, or deleting the entire current line with the kill character. Other special characters generate signals sent to processes associated with the keyboard; these signals may abort processing or may suspend it. Additional characters start and stop output, flush output, or prevent special interpretation of the succeeding character. The user can type several lines of input, up to an implementation-defined limit, without waiting for input to be read and then removed from the input queue. The user can specify the special processing characters or can selectively disable them.

Screen editors and programs that communicate with other computers generally run in noncanonical mode (also commonly referred to as raw mode or character-at-a-time mode). In this mode, the system makes each typed character available to be read as input as soon as that character is received. All special-character input processing is disabled, no erase or other line-editing processing is done, and all characters are passed to the program reading from the keyboard.

It is possible to configure keyboard input in thousands of combinations between these two extremes. For example, a screen editor that wanted to receive user interrupts asynchronously might enable the special characters that generate signals but otherwise run in noncanonical mode.

In addition to processing input characters, the terminal interface must do certain processing on output. Most of the time, this processing is simple: Newline characters are converted to a carriage return plus a line feed. In addition to doing character processing, the terminal output routines must manage flow control, both with the user (using stop and start characters) and with the process. Because users absorb output slowly in comparison with computer peripherals, a program writing to a terminal may produce output much faster than that output can be handled by the user. When a process has filled the terminal output queue, it will be put to sleep; it will be restarted when enough output has drained.


10.2. Line Disciplines

Most of the character processing done for terminal interfaces is independent of whether it is associated with a pseudo-terminal or a real hardware device. Therefore, most of this processing is done by common routines in the tty driver, or terminal handler. A hardware interface is supported by a specific device driver. The hardware driver is a device driver like those described in Chapter 6. It is responsible for receiving and transmitting characters and for handling some of the synchronization with the process doing output. The hardware driver is called by the tty driver to do output; in turn, it calls the tty driver with input characters as they are received. The pseudo-terminal interface acts as a software emulation of an asynchronous serial interface. Thus, a pseudo-terminal is indistinguishable from real hardware to the layers above and below it in the kernel.

The tty driver interfaces with the rest of the system as a line discipline. A line discipline is a processing module used to provide semantics on an asynchronous serial interface. It is described by a procedural interface, the linesw (line-switch) structure. The linesw structure specifies the entry points of a line discipline, much as the character-device switch cdevsw lists the entry points of a character device driver (see Section 6.2). The entry points of a line discipline are listed in Table 10.1 (on page 416). Like all device drivers, a terminal driver is divided into the top half, which runs synchronously when called to process a system call, and the bottom half, which runs asynchronously when characters are presented to it from a pseudo-terminal or hardware device. The line discipline provides routines that do common terminal processing for both the top and bottom halves of a terminal driver.

Table 10.1. Entry points of a line discipline.

Routine

Called from

Usage

l_open

above

initial entry to discipline

l_close

above

exit from discipline

l_read

above

read from line

l_write

above

write to line

l_ioctl

above

control operations

l_rint

below

received character

l_start

below

completion of transmission

I_modem

below

modem carrier transition

The serial terminal interface supports the normal set of character device-driver entry points specified by the character-device switch. Several of the standard driver entry points (read, write, and ioctl) immediately transfer control to the line discipline when called. (The standard tty polling routine ttypoll() usually is used as the device driver poll entry in the character-device switch.) The open and close routines are similar; the line-discipline open entry is called when a line first enters a discipline, either at initial open of the line or when the discipline is changed. Similarly, the discipline close routine is called to exit from a discipline. All these routines are called from above in response to a corresponding system call. The remaining line-discipline entries are called by the bottom half of the device driver to report input or status changes detected at interrupt time. The l_rint (receiver interrupt) entry is called with each character received on a line. The corresponding entry for transmit-complete interrupts is the l_start routine, which is called when output operations complete. This entry gives the line discipline a chance to start additional output operations. For the normal terminal line discipline, this routine simply calls the driver's output routine to start the next block of output. Transitions in modem-control lines (see Section 10.7) may be detected by the hardware driver, in which case the l_modem routine is called with an indication of the new state.

The system includes several different types of line disciplines. Most lines use the terminal-oriented discipline described in Section 10.3. Other disciplines in the system support various asynchronous serial network interface protocols.


10.3. User Interface

The default terminal line discipline is derived from a discipline that was present in System V, as modified by the POSIX standard and then was modified further to provide reasonable compatibility with previous 4.2BSD line disciplines. The base structure used to describe terminal state in System V was the termio structure. The base structure used by POSIX and by FreeBSD is the termios structure.

The standard programmatic interface for control of the terminal line discipline is the ioctl system call. This call changes disciplines, sets and gets values for special processing characters and modes, sets and gets hardware serial line parameters, and performs other control operations. Most ioctl operations require one argument in addition to a file descriptor and the command; the argument is the address of an integer or structure from which the system gets parameters or into which information is placed. Because the POSIX Working Group thought that the ioctl system call was difficult and undesirable to specify—because of its use of arguments that varied in size, in type, and in whether they were being read or written—the group members chose to introduce new interfaces for each of the ioctl calls that they believed were necessary for application portability. Each of these calls is named with a tc prefix. In the FreeBSD system, each of these calls is translated (possibly after preprocessing) into an ioctl call.

The following set of ioctl commands apply specifically to the standard terminal line discipline, although all line disciplines must support at least the first two. Other disciplines generally support other ioctl commands. This list is not exhaustive, although it presents all the commands that are used commonly.

TIOCGETD TIOCSETD | Get (set) the line discipline for this line.
TIOCGETA TIOCSETA | Get (set) the termios parameters for this line, including line speed, behavioral parameters, and special characters (e.g., erase and kill characters).
TIOCSETAW | Set the termios parameters for this line after waiting for the output buffer to drain (but without discarding any characters from the input buffer).
TIOCSETAF | Set the termios parameters for this line after waiting for the output buffer to drain and discarding any characters from the input buffer.
TIOCFLUSH | Discard all characters from the input and output buffers.
TIOCDRAIN | Wait for the output buffer to drain.
TIOCEXCL TIOCNXCL | Get (release) exclusive use of the line.
TIOCCBRK TIOCSBRK | Clear (set) the terminal hardware BREAK condition for the line.
TIOCCDTR TIOCSDTR | Clear (set) data terminal ready on the line.
TIOCGPGRP TIOCSPGRP | Get (set) the process group associated with this terminal (see Section 10.5).
TIOCOUTQ | Return the number of characters in the terminal's output buffer.
TIOCSTI | Enter characters into the terminal's input buffer as though they were typed by the user.
TIOCNOTTY | Disassociate the current controlling terminal from the process (see Section 10.5).
TIOCSCTTY | Make the terminal the controlling terminal for the process (see Section 10.5).
TIOCSTART TIOCSTOP | Start (stop) output on the terminal.
TIOCGWINSZ TIOCSWINSZ | Get (set) the terminal or window size for the terminal line; the window size includes width and height in characters and (optionally, on graphical displays) in pixels.


10.4. The tty Structure

Each hardware or pseudo-terminal has a data structure to contain its state. This structure, the tty structure (see Table 10.2), contains state information, the input and output queues, the modes and options set by the ioctl operations listed in Section 10.3, and the line-discipline number. The tty structure is shared by the pseudo-terminal or hardware driver and the line discipline. The calls to the line discipline all require a tty structure as a parameter; the driver locates the correct tty according to its identity.

Table 10.2. The tty structure.

Type

Description

character queues

raw input queue canonical input queue device output queue high/low watermarks

hardware parameters

device identifier start/stop output functions set hardware state function

polling

thread polling for reading thread polling for writing

state

termios state process group session terminal column number number of rows and columns

The sections of the tty structure include the following:

  • State information about the terminal. The t_state field includes major file options (e.g., signal-driven I/O) and transient state for flow control and synchronization. For hardware-based terminals, it includes line state (open, carrier present, or waiting for carrier).

  • Input and output queues. The output driver transmits characters placed in the output queue, t_outq. Line disciplines generally use the t_rawq and t_canq (noncanonical and canonical queues) for input; in line mode, the canonical queue contains full lines, and the noncanonical queue contains any current partial line. In addition, t_hiwat provides the boundary where threads attempting to write to the terminal will be put to sleep, waiting for the output queue to drain; t_lowat provides the boundary where sleeping threads waiting to write to the terminal will be awakened.

  • Hardware and software modes and parameters, and special characters. The t_termios structure contains the information set by TIOCSETA, TIOCSETAF, and TIOCSETAW. Specifically, line speed appears in the c_ispeed and c_ospeed fields of the t_termios structure, control information in the c_iflag, c_oflag, c_cflag and c_lflag fields, and special characters (end-of-file, end-of-line, alternate end-of-line, erase, word-erase, kill, reprint, interrupt, quit, suspend, start, stop, escape-next-character, status-interrupt, flush-output and VMIN and VTIME information) in the c_cc field.

  • Hardware driver information. This information includes t_oproc and t_stop, the driver procedures that start and stop transmissions after data are placed in the output queue; t_param, the driver procedure that sets the hardware state; and t_kdev, the device associated with the terminal line.

  • Terminal line-discipline software state. This state includes the terminal column number and counts for tab and erase processing (t_column, t_rocount, and t_rocol), the process group of the terminal (t_pgrp), the session associated with the terminal (t_session), and information about any processes selecting for input or output (t_rsel and t_wsel).

  • Terminal or window size (t_winsize). This information is not used by the kernel, but it is stored here to present consistent and correct information to applications. In addition, FreeBSD supplies the SIGWINCH signal (derived from Sun Microsystems' SunOS) that can be sent when the size of a window changes. This signal is useful for windowing packages such as X Window System [Scheifler & Gettys, 1986] that allow users to resize windows dynamically; programs such as text editors running in such a window need to be informed that something has changed and that they should recheck the window size.

The tty structure is initialized by the terminal driver's open routine and by the line-discipline open routine.


10.5. Process Groups, Sessions, and Terminal Control

The process-control (job-control) facilities, described in Section 4.8, depend on the terminal I/O system to control access to the terminal. Each job (a process group that is manipulated as a single entity) is known by a process-group ID.

Each terminal structure contains a pointer to an associated session. When a process creates a new session, that session has no associated terminal. To acquire an associated terminal, the session leader must make an ioctl system call using a file descriptor associated with the terminal and specifying the TIOCSCTTY flag. When the ioctl succeeds, the session leader is known as the controlling process. In addition, each terminal structure contains the process group ID of the foreground process group. When a session leader acquires an associated terminal, the terminal process group is set to the process group of the session leader. The terminal process group may be changed by making an ioctl system call using a file descriptor associated with the terminal and specifying the TIOCSPGRP flag. Any process group in the session is permitted to become the foreground process group for the terminal.

Signals that are generated by characters typed at the terminal are sent to all the processes in the terminal's foreground process group. By default, some of those signals cause the process group to stop. The shell creates jobs as process groups, setting the process group ID to be the PID of the first process in the process group. Each time it places a new job in the foreground, the shell sets the terminal process group to the new process group. Thus, the terminal process group is the identifier for the process group that is currently in control of the terminal— that is, for the process group running in the foreground. Other process groups may run in the background. If a background process attempts to read from the terminal, its process group is sent another signal, which stops the process group. Optionally, background processes that attempt terminal output may be stopped as well. These rules for control of input and output operations apply to only those operations on the controlling terminal.

When carrier is lost for the terminal—for example, at modem disconnect or when a network connection is lost—the session leader of the session associated with the terminal is sent a SIGHUP signal. If the session leader exits, the controlling terminal is revoked, and that invalidates any open file descriptors in the system for the terminal. This revocation ensures that processes holding file descriptors for a terminal cannot access the terminal after the terminal is acquired by another user. The revocation operates at the vnode layer. It is possible for a process to have a read or write sleeping for some reason—for example, it was in a background process group. Since such a process would have already resolved the file descriptor through the vnode layer, a single read or write by the sleeping process could complete after the revoke system call. To avoid this security problem, the system checks a tty generation number when a process wakes up from sleeping on a terminal and, if the number has changed, restarts the read or write system call.


10.6. C-lists

The terminal I/O system deals with data in blocks of widely varying sizes. Most input and output operations deal with single characters (typed input characters and their output echoes). Input characters are usually aggregated with previous input to form lines of varying sizes. Some output operations involve larger amounts of data, such as screen updates or other command output. The data structures originally designed for terminal drivers, the character block, C-block, and character list, C-list, are still in use in FreeBSD. Each C-block is a fixed-size buffer that contains a linkage pointer and space for buffered characters and quoting information. Its size is a power of 2, and it is aligned such that the system can compute boundaries between blocks by masking off the low-order bits of a pointer. FreeBSD uses 128-byte C-blocks, storing 108 characters and an array of quoting flags (1-bit per character). A queue of input or output characters is described by a C-list, which contains pointers to the first and final characters and a count of the number of characters in the queue (see Figure 10.1). Both of the pointers point to characters stored in C-blocks. When a character is removed from a C-list queue, the count is decremented, and the pointer to the first character is incremented. If the pointer has advanced beyond the end of the first C-block on the queue, the pointer to the next C-block is obtained from the forward pointer at the start of the current C-block. After the forward pointer is updated, the empty C-block is placed on a free chain. A similar process adds a character to a queue. If there is no room in the current buffer, another buffer is allocated from the free list, the linkage pointer of the last buffer is set to point at the new buffer, and the tail pointer is set to the first storage location of the new buffer. The character is stored at the location pointed to by the tail pointer, the tail pointer is incremented, and the character count is incremented. A set of utility routines manipulates C-lists: getc() removes the next character from a C-list and returns that character and putc() adds a character to the end of a C-list. The getc() routine returns an integer, and the putc() routine takes an integer as an argument. The lower 8 bits of this value are the actual character. The upper bits are used to provide quoting and other information. Groups of characters may be added to or removed from C-lists with b_to_q() and q_to_b(), respectively, in which case no additional information (e.g., quoting information) can be specified or returned. The terminal driver also requires the ability to remove a character from the end of a queue with unputc(), to examine characters in the queue with nextc(), and to concatenate queues with catq().

Figure 10.1. A C-list structure.

When UNIX was developed on computers with small address spaces, the design of buffers for the use of terminal drivers was a challenge. The C-list and C-block provided an elegant solution to the problem of storing arbitrary-length queues of data for terminal input and output queues when the latter were designed for machines with small memories. On modern machines that have far larger address spaces, it would be better to use a data structure that uses less CPU time per character at a cost of reduced space efficiency. FreeBSD still uses the original C-list data structure because of the high labor cost of converting to a new data structure; a change to the queue structure would require changes to all the line disciplines and to all the terminal device drivers, which would be a substantial amount of work. The developers could change just the implementations of the interface routines, but the routines would still be called once per character unless the actual interface was changed, and changing the interface would require changing the drivers.


10.7. RS-232 and Modem Control

Most terminals and modems are connected via asynchronous RS-232 serial ports. This type of connection supports several lines, in addition to those that transmit and receive data. The system typically supports only a few of these lines. The most commonly used lines are those showing that the equipment on each end is ready for data transfer. The RS-232 electrical specification is asymmetrical: Each line is driven by one of the two devices connected and is sampled by the other device. Thus, one end in any normal connection must be wired as data-terminal equipment (DTE), such as a terminal, and the other as data-communications equipment (DCE), such as a modem. Note that terminal in DTE means endpoint: A terminal on which people type is a DTE, and a computer also is a DTE. The data-terminal-ready (DTR) line is the output of the DTE end that serves as a ready indicator. In the other direction, the data-carrier-detect (DCD) line indicates that the DCE device is ready for data transfer. Historically, VAX terminal interfaces were all wired as DTE (they may be connected directly to modems or connected to local terminals with null modem cables). The terminology used in the FreeBSD terminal drivers and commands reflects this orientation, even though many computers incorrectly use the opposite convention.

When terminal devices are opened, the DTR output is asserted so that the connected modem or other equipment may begin operation. If modem control is supported on a line, the open does not complete unless the O_NONBLOCK option was specified or the CLOCAL control flag is set for the line, and no data are transferred until the DCD input carrier is detected or the CLOCAL flag is set. Thus, an open on a line connected to a modem will block until a connection is made; the connection commonly occurs when a call is received from a remote modem. Data then can be transferred for as long as carrier remains on. If the modem loses the connection, the DCD line is turned off and later reads and writes fail.

Ports that are used with local terminals or other DTE equipment are connected with a null-modem cable that connects DTR on each end to DCD on the other end. Alternatively, the DTR output on the host port can be looped back to the DCD input. If the cable or device does not support modem control, the system will ignore the state of the modem control signals when the CLOCAL control flag is set for the line. Finally, some drivers may be configured to ignore modem-control inputs.


10.8. Terminal Operations

Now that we have examined the overall structure of the terminal I/O system and have described that system's data structures, as well as the hardware that the system controls, we continue with a description of the terminal I/O system operation. We examine the operation of the pseudo-terminal device driver and the default termios line discipline.

Open

Each time that the master side of a previously unused pseudo-terminal device is opened, the pseudo-terminal driver's open routine is called. The open routine initializes the tty structure; sets the default modes; sets the terminal-size information for the line to zero, indicating an unknown size; and sets the state to TS ISOPEN. The driver then passes control to the line discipline through its open entry. The setup of the terminal completes when the corresponding slave side of the pseudo-terminal is opened by the application that will be using it.

For hardware devices that support modem-control lines, the open routine takes some additional steps. It starts by enabling the DTR output line. If the CLOCAL control flag is not set for the terminal and the open call did not specify the O_NONBLOCK flag, the open routine blocks awaiting assertion of the DCD input line. Some drivers support device flags to override modem control; these flags are set in the system-configuration file and are stored in the driver data structures. If the bit corresponding to a terminal line number is set in a device's flags, modem-control lines are ignored on input. When a carrier signal is detected on the line, the TS_CARR_ON bit is set in the terminal state. The line is then marked as open (state bit TS_OPEN), and the driver then passes control to the line discipline as just described for pseudo-terminals.

Output Line Discipline

After a line has been opened, a write on the resulting file descriptor produces output to be transmitted on the terminal. Writes to character devices result in calls to the device write entry, d_write, with a device pointer, a uio structure describing the data to be written, and a flag specifying whether the I/O is nonblocking. The pseudo-terminal driver uses the device pointer to locate the correct tty structure and then call the line discipline l_write entry with the tty structure and uio structure as parameters.

The line-discipline write routine does most of the work of output translation and flow control. It is responsible for copying data into the kernel from the user process and for placing the translated data onto the pseudo-terminal's output queue. The terminal-driver write routine, ttwrite(), first checks whether the current process is allowed to write to the terminal at this time. The user may set a tty option to allow only the foreground process (see Section 10.5) to do output. If this option is set, and if the terminal line is the controlling terminal for the process, then the process should do output immediately only if it is in the foreground process group (i.e., if the process groups of the process and of the terminal are the same). If the process is not in the foreground process group, and a SIGTTOU signal would cause the process to be suspended, a SIGTTOU signal is sent to the process group of the process. Here, the write will be attempted again when the user moves the process group to the foreground. If the process is in the foreground process group, or a SIGTTOU signal would not suspend the process, the write proceeds as usual.

When ttwrite() has confirmed that the write is permitted, it enters a loop that copies the data to be written into the kernel, checks for any output translation that is required, and places the data on the output queue for the terminal. It prevents the queue from becoming overfull by blocking if the queue fills before all characters have been processed. The limit on the queue size, the high watermark, is dependent on the output line speed; for pseudo-terminals the line speed is set to the maximum baud rate so that they will get the maximum high watermark of several thousand characters. The low watermark is set to about half of the high watermark. When forced to wait for output to drain before proceeding, ttwrite() sets a flag in the tty structure state, TS_SO_OLOWAT, to request that it be awakened when the queue drops below the low watermark. The check of the queue size and following sleep must be ordered such that removal of characters from the output queue is guaranteed to occur after the sleep; see Figure 10.2. If characters could be removed after the check but before the sleep, their removal might cause the queue to drop below the low watermark, thus causing a wakeup to be issued before the thread had gone to sleep. Once asleep, the thread would be blocked forever because the event for which it would be waiting would have already occurred. As shown in Figure 10.2, the guarantee is made by acquiring the mutex that must be held while adding or removing characters from the output queue for the terminal.

Figure 10.2. Pseudocode for checking the output queue in a line discipline.
struct tty *tp;

ttstart(tp);
mtx_lock(&tp–>t_outmtx);
if (tp–>t_outq.c_cc > high-water-mark) {
    /* Request wake up when output <= low-water-mark */
    tp–>t_state |= TS_SO_OLOWAT;
    msleep(&tp–>t_outq, &tp–>t_outmtx, TTOPRI, "ttywri", 0);
}
mtx_unlock(&tp–>t_outmtx);

Once errors, permissions, and flow control have been checked, ttwrite() copies the user's data into a local buffer in chunks of at most 100 characters using uiomove(). (A value of 100 is used because the buffer is stored on the stack, and so it cannot be large.) When the terminal driver is configured in noncanonical mode, no per-character translations are done, and the entire buffer is processed at once. In canonical mode, the terminal driver locates groups of characters requiring no translation by scanning through the output string, looking up each character in turn in a table that marks characters that might need translation (e.g., newline), or characters that need expansion (e.g., tabs). Each group of characters that requires no special processing is placed into the output queue using b_to_q(). Trailing special characters are output with ttyoutput(). In either case, ttwrite() must check that enough C-list blocks are available; if they are not, it waits for a short time (by sleeping on Ibolt for up to 1 second), then retries.

The routine that does output with translation is ttyoutput(), which accepts a single character, processes that character as necessary, and places the result on the output queue. The following translations may be done, depending on the terminal mode:

  • Tabs may be expanded to spaces.

  • Newlines may be replaced with a carriage return plus a line feed.

As soon as data are placed on the output queue of a tty, ttstart() is called to start output. Unless output is already in progress or has been suspended by receipt of a stop character, ttstart() calls the start routine specified in the tty's t_oproc field. For a pseudo-terminal, the start routine checks for a thread sleeping on the master side and, if found, awakens it so that it can consume the data. For a hardware terminal, the start routine begins sending the characters out the serial line. Once all the data have been processed and have been placed into the output queue, ttwrite() returns an indication that the write completed successfully.

Terminal Output

Characters are removed from the output queue either by the thread running on the master side of the pseudo-terminal or by the hardware device driver. Whenever the number of characters on the output queue drops below the low watermark, the output routine checks to see if the TS_SO_OLOWAT flag is set to show that a thread is waiting for space in the output queue and should be awakened. In addition, selwakeup() is called, and if a thread is recorded in t_wsel as selecting for output, that thread is notified. The output continues until the output queue is empty.

Terminal Input

Unlike output, terminal input is not started by a system call but instead arrives asynchronously when the terminal line receives characters from a remote login session or locally from the keyboard. Thus, the input processing in the terminal system occurs mostly at interrupt time.

When a character arrives over the network from a remote login session, the locally running remote-login daemon writes it into the master side of the pseudo-terminal. The master side of the pseudo-terminal will pass the character as input to the terminal line discipline for the receiving tty through the latter's l_rint entry:

(*linesw[tp–>t_line].l_rint)(input-character, tp);

For locally attached hardware such as a keyboard, the input character will be passed by the device driver directly to the receiving tty's l_rint entry. In either case, the input character is passed to the l_rint routine as an integer. The bottom 8 bits of the integer are the actual character. Characters received from locally connected hardware may have hardware-detected parity errors, break characters, or framing errors. Such errors are shown by setting flags in the upper bits of the integer.

The receiver-interrupt (l_rint) routine for the normal terminal line discipline is ttyinput(). When a break condition is detected (a longer-than-normal character with only 0 bits), it is ignored, or an interrupt character or a null is passed to the process, depending on the terminal mode. The interpretation of terminal input described in Section 10.1 is done here. Input characters are echoed if desired. In noncanonical mode, characters are placed into the raw input queue without interpretation. Otherwise, most of the work done by ttyinput() is to check for characters with special meanings and to take the requested actions. Other characters are placed into the raw queue. In canonical mode, if the received character is a carriage return or another character that causes the current line to be made available to the program reading the terminal, the contents of the raw queue are added to the canonicalized queue and ttwakeup() is called to notify any process waiting for input. In noncanonical mode, ttwakeup() is called when each character is processed. It will awaken any process sleeping on the raw queue awaiting input for a read and will notify processes selecting for input. If the terminal has been set for signal-driven I/O using fcntl and the FASYNC flag, a SIGIO signal is sent to the process group controlling the terminal.

Ttyinput() must also check that the input queue does not become too large, exhausting the supply of C-list blocks. For pseudo-terminals, when the input queue becomes full, the kernel simply stops reading characters from the master side, which causes the local-login daemon to stop reading from the network. Eventually the network flow control blocks the process generating the characters. For characters coming from locally attached hardware such as a keyboard, input characters are discarded when the limit (8192 characters) is reached. If the IXOFF termios flag is set, end-to-end flow control is invoked when the queue reaches half full by output of a stop character (normally XOFF or control-S).

Up to this point, all processing is asynchronous and occurs independent of whether a read call is pending on the terminal device. In this way, type-ahead is allowed to the limit of the input queues.

Eventually, a read call is made on the file descriptor for the terminal device. Like all calls to read from a character-special device, this one results in a call to the device driver's d_read entry with a device pointer, a uio structure describing the data to be read, and a flag specifying whether the I/O is nonblocking. Terminal device drivers use the device pointer to locate the tty structure for the device and then call the line discipline l_read entry to process the system call.

The l_read entry for the terminal driver is ttread(). Like ttwrite(), ttread() first checks to see whether the process is part of the session and the process group currently associated with the terminal. If the process is a member of the session currently associated with the terminal, if any, and is a member of the current process group, the read proceeds. Otherwise, if a SIGTTIN would suspend the process, a SIGTTIN is sent to that process group. Here, the read will be attempted again when the user moves the process group to the foreground. Otherwise, an error is returned. Finally, ttread() checks for data in the appropriate queue (the canonical queue in canonical mode, the raw queue in noncanonical mode). If no data are present, ttread() returns the error EAGAIN if the terminal is using non-blocking I/O; otherwise, it sleeps on the address of the raw queue. When ttread() is awakened, it restarts processing from the beginning because the terminal state or process group might have changed while it was asleep.

When characters are present in the queue for which ttread() is waiting, they are removed from the queue one at a time with getc() and are copied out to the user's buffer with ureadc(). In canonical mode, certain characters receive special processing as they are removed from the queue: The delayed-suspension character causes the current process group to be stopped with signal SIGTSTP, and the end-of-file character terminates the read without being passed back to the user program. If there was no previous character, the end-of-file character results in the read returning zero characters, and that is interpreted by user programs as indicating end-of-file. However, most special processing of input characters is done when the character is entered into the queue. For example, translating carriage returns to newlines based on the ICRNL flag must be done when the character is first received because the newline character wakes up waiting processes in canonical mode. In noncanonical mode, the characters are not examined as they are processed.

Characters are processed and returned to the user until the character count in the uio structure reaches zero, the queue is exhausted, or, if in canonical mode, a line terminator is reached. When the read call returns, the returned character count will be the amount by which the requested count was decremented as characters were processed.

After the read completes, if terminal output was blocked by a stop character being sent because the queue was filling up, and the queue is now less than 20 percent full, a start character (normally XON, control-Q) is sent.

The ioctl Routine

Section 10.3 described the user interface to terminal drivers and line disciplines, most of which is accessed via the ioctl system call. Most of these calls manipulate software options in the terminal line discipline; some of them also affect the operation of the asynchronous serial-port hardware. In particular, the hardware line speed, word size, and parity are derived from these settings. Consequently, ioctl calls are processed both by the current line discipline and by the hardware driver.

The device driver d_ioctl routine is called with a device pointer, an ioctl command, and a pointer to a data buffer when an ioctl is done on a character-special file, among other arguments. Like the read and write routines, most terminal-driver ioctl routines locate the tty structure for the device and then pass control to the line discipline. The line-discipline ioctl routine does discipline-specific actions, including change of line discipline. If the line-discipline routine fails, the driver will immediately return an error, as shown in Figure 10.3. Otherwise, the driver will then call the ttioctl() routine that does most common terminal processing, including changing terminal parameters. If ttioctl() fails, the driver will immediately return an error. Otherwise, some drivers implement additional ioctl commands that do hardware specific processing—for example, manipulating modem-control outputs. These commands are not recognized by the line discipline or by common terminal processing, and thus must be handled by the driver. The ioctl routine returns an error number if an error is detected or returns zero if the command has been processed successfully. The errno variable is set to ENOTTY if the command is not recognized.

Figure 10.3. Handling of an error return from a line discipline.
error = (*linesw[tp–>t_line].l_ioctl)(tp, cmd, data, flag);
if (error >= 0)
    return (error);
error = ttioctl(tp, cmd, data, flag);
if (error >= 0)
    return (error);
switch (cmd)  {
case TIOCSBRK:      /* hardware specific commands */
    ...
    return (0);
case TIOCCBRK:
    ...
    return (0);
default:
    return (ENOTTY);
}

Modem Transitions

The way in which the system uses modem-control lines on terminal lines was introduced in Section 10.7. Most terminal hardware support at least the set of modem-control lines used by FreeBSD; those that do not act instead as though the carrier were always asserted. When a device is opened, the DTR output is enabled, and then the state of the carrier input is checked. If the state of the carrier input changes later, this change must be detected and processed by the driver. Some devices have a separate interrupt that reports changes in modem-control status; others report such changes along with other status information with received characters. Some devices do not interrupt when modem-control lines change, and the driver must check their status periodically. When a change is detected, the line discipline is notified by a call to its l_mnodem routine with the new state of the carrier input.

The normal terminal-driver modem routine, ttymodem(), maintains the state of the TS_CARR_ON flag in the tty structure and processes corresponding state changes. When carrier establishment is detected, a wakeup is issued for any process waiting for an open to complete. When carrier drops on an open line, the leader of the session associated with the terminal (if any) is sent a hangup signal, SIGHUP, and the terminal queues are flushed. The return value of ttymodem() indicates whether the driver should maintain its DTR output. If the value is zero, DTR should be turned off. Ttymodem() also implements an obscure terminal option to use the carrier line for flow-control handshaking, stopping output when carrier drops and resuming when it returns.

Closing of Terminal Devices

When the final reference to a terminal device is closed, or the revoke system call is made on the device, the device-driver close routine is called. Both the line discipline and the hardware driver may need to close down gracefully. The devicedriver routine first calls the line-discipline close routine. The standard linediscipline close entry, ttylclose(), waits for any pending output to drain (if the terminal was not opened with the O_NONBLOCK flag set and the carrier is still on), then flushes the input and output queues. (Note that the close may be interrupted by a signal while waiting for output to complete.) The hardware driver may clear any pending operations, such as transmission of a break. If the state bit TS_HUPCLS has been set with the TIOCHPCL ioctl, DTR is disabled to hang up the line. Finally, the device-driver routine calls ttyclose(), which flushes all the queues, increments the generation number so that pending reads and writes can detect reuse of the terminal, and clears the terminal state.


10.9. Other Line Disciplines

We have examined the operation of the terminal I/O system using the standard terminal-oriented line-discipline routines. For completeness, we will briefly describe the other class of line disciplines in the system used to connect serial lines to the network.

The point-to-point protocol (PPP), point-to-point line discipline is used by networking software to encapsulate and transfer Internet Protocol (IP) datagrams over asynchronous serial lines [Simpson, 1994]. (See Chapter 13 for information about IP.) The PPP line discipline allows moderate-speed network connections from a machine lacking high-speed network hardware.

The ppp program opens a serial line, sets the line's speed, and enters the PPP line discipline. The PPP line discipline's open routine associates the terminal line with a preconfigured network interface and prepares to send and receive network packets. Once the interface's network address is set with the ifconflg program, the network will route packets through the PPP line to the system to which it connects.

The output path is started every time a packet is output to the PPP interface. Packets are enqueued on one of two queues: one for interactive traffic and one for other traffic. Interactive traffic takes precedence over other traffic. The PPP discipline places the framing characters and the data of the next packet onto the output queue of the tty, sometimes compressing packet headers. It then starts transmission by calling ttstart(), which in turn calls the device's start routine referenced in the tty t_oproc field. It may place multiple packets onto the output queue before returning, as long as the system is not running short of C-list blocks. However, it stops moving packets into the tty output queue when the character count has reached a low limit (60 bytes) so that future interactive traffic is not blocked by noninteractive traffic already in the output queue. When transmission completes, the device driver calls the PPP start routine, which continues to place data onto the output queue until all packets have been sent or the queue hits the limit again.

When characters are received on a line that is using the PPP discipline, the data characters are placed into a network buffer. When a framing character ends the packet, the packet header is uncompressed if necessary, the packet is presented to the network protocol, and the buffer is reinitialized.


Exercises

10.1 What are the two general modes of terminal input? Which mode is most commonly in use when users converse with an interactive screen editor?

10.2 Explain why there are two character queues for dealing with terminal input. Describe the use of each.

10.3 What do we mean when we say that modem control is supported on a terminal line? How are terminal lines of this sort typically used?

10.4 What signal is sent to what process associated with a terminal if a user disconnects the modem line in the middle of a session?

10.5 Name the two devices that make up a pseudo-terminal. Explain the role of each of these pieces.

10.6 How is the high watermark on a terminal's output queue determined?

*10.7 Consider a facility that allowed a tutor on one terminal to monitor and assist students working on other terminals. Everything the students typed would be transmitted both to the system as input and to the tutor's terminal as output. Everything the tutor typed would be directed to the students' terminals as input. Describe how this facility might be implemented with a special-purpose line discipline. Describe further useful generalizations of this facility.

*10.8 The terminal line discipline supports logical erasure of input text when characters, words, and lines are erased. Remembering that other system activities continue while a user types an input line, explain what complications must be considered in the implementation of this feature. Name three exceptional cases, and describe their effects on the implementation.

**10.9 Propose another buffering scheme to replace C-lists.


References

Scheifler & Gettys, 1986. R. W. Scheifler & J. Gettys, "The X Window System," ACM Transactions on Graphics, vol. 5, no. 2, pp. 79–109, April 1986.

Simpson, 1994. W. Simpson, "The Point-to-Point Protocol (PPP)," RFC 1661, available from http://www.faqs.org/rfcs/rfcl661.html, July 1994.


Part IV: Interprocess Communication

    Chapter 11. Interprocess Communication

    Chapter 12. Network Communication

    Chapter 13. Network Protocols


Chapter 11. Interprocess Communication

FreeBSD provides a rich set of interprocess-communication facilities intended to support the construction of distributed and multiprocess programs built on top of communications primitives. Support for these facilities is described in this chapter.

No one mechanism can provide for all types of interprocess communication. The subsystems that provide IPC in FreeBSD 5.2 can be broken down into two areas. The first provides for IPC on a single system and includes support for semaphores, message queues, and shared memory. The second is the socket interface, which provides a uniform communication API for network communication.

The socket API is deeply entwined with the network subsystem. The architecture of the network system is described in Chapter 12, and the networking protocols themselves are examined in Chapter 13. You will find it easiest to understand the material in these three chapters if you read this chapter before you read Chapters 12 and 13. At the end of Chapter 13 is a section devoted to tying everything together.


11.1. Interprocess-Communication Model

There were several goals in the design of the interprocess-communication enhancements to UNIX. The most immediate need was to provide access to communication networks such as the Internet [Cerf, 1978]. Previous work in providing network access had focused on the implementation of the network protocols, exporting the transport facilities to applications via special-purpose—and often awkward—interfaces [Cohen, 1977; Gurwitz, 1981]. As a result, each new network implementation resulted in a different application interface, requiring most existing programs to be altered significantly or rewritten completely. For 4.2BSD the interprocess-communication facilities were intended to provide a sufficiently general interface to allow network-based applications to be constructed independently of the underlying communication facilities.

The second goal was to allow multiprocess programs, such as distributed databases, to be implemented. The UNIX pipe requires all communicating processes to be derived from a common parent process. The use of pipes forced systems to be designed with a somewhat contorted structure. New communication facilities were needed to support communication between unrelated processes residing locally on a single host computer and residing remotely on multiple host machines.

Finally, it became important to provide new communication facilities to allow construction of local-area network services, such as file servers. The intent was to provide facilities that could be used easily in supporting resource sharing in a distributed environment and not to build a distributed UNIX system.

The interprocess-communication facilities were designed to support the following:

  • Transparency: Communication between processes should not depend on whether the processes are on the same machine.

  • Efficiency: The applicability of any interprocess-communication facility is limited by the performance of the facility. A naive implementation of interprocess-communication often results in a very modular but inefficient implementation because most interprocess communication facilities, especially those related to networks, are broken down into many layers. At each layer boundary the software must perform some work, either adding information to a message or removing it. FreeBSD only introduces layers where they are absolutely necessary for the proper functioning of the system and does not introduce arbitrary layers where they are not necessary.

  • Compatibility: Existing naive processes should be usable in a distributed environment without change. A naive process is characterized as a process that performs its work by reading from the standard input file and writing to the standard output file. A sophisticated process uses knowledge about the richer set of interfaces provided by the Operating System to do its work. A major reason why UNIX has been successful is the operating system's support for modularity by naive processes that act as byte-stream filters. Although sophisticated applications such as Web servers and screen editors exist, they are far outnumbered by the collection of naive application programs.

While designing the interprocess-communication facilities, the developers identified the following requirements to support these goals, and they developed a unifying concept for each:

  • The system must support communication networks that use different sets of protocols, different naming conventions, different hardware, and so on. The notion of a communication domain was defined for these reasons. A communication domain embodies the standard semantics of communication and naming. Different networks have different standards for naming communication end-points, which may also vary in their properties. In one network, a name may be a fixed address for a communication endpoint, whereas in another it may be used to locate a process that can move between locations. The semantics of communication can include the cost associated with the reliable transport of data, the support for multicast transmissions, the ability to pass access rights or capabilities, and so on.

  • A unified abstraction for an endpoint of communication is needed that can be manipulated with a file descriptor. The socket is the abstract object from which messages are sent and received. Sockets are created within a communication domain, just as files are created within a filesystem. Unlike files, however, sockets exist only as long as they are referenced.

  • The semantic aspects of communication must be made available to applications in a controlled and uniform way. Applications must be able to request different styles of communication, such as reliable byte stream or unreliable datagram, and these styles must be provided consistently across all communication domains. All sockets are typed according to their communication semantics. Types are defined by the semantic properties that a socket supports. These properties are

    1. In-order delivery of data

    2. Unduplicated delivery of data

    3. Reliable delivery of data

    4. Connection-oriented communication

    5. Preservation of message boundaries

    6. Support for out-of-band messages

Pipes have the first four properties, but not the fifth or sixth. An out-of-band message is one that is delivered to the receiver outside the normal stream of incoming, in-band data. It usually is associated with an urgent or exceptional condition. A connection is a mechanism that protocols use to avoid having to transmit the identity of the sending socket with each packet of data. Instead, the identity of each endpoint of communication is exchanged before transmission of any data and is maintained at each end so that it can be presented at any time. On the other hand, connectionless communications require a source and destination address associated with each transmission. A datagram socket provides unreliable, connectionless packet communication; a stream socket provides a reliable, connection-oriented byte stream that may support out-of-band data transmission; and a sequenced packet socket provides a sequenced, reliable, unduplicated connection-based communication that preserves message boundaries. Other types of sockets are desirable and can be added.

Processes must be able to locate endpoints of communication so that they can rendezvous without being related, so sockets can be named. A socket's name is meaningfully interpreted only within the context of the communication domain in which the socket is created. The names used by most applications are human-readable strings. However, the name for a socket that is used within a communication domain is usually a low-level address. Rather than placing name-to-address translation functions in the kernel, FreeBSD 5.2 provides functions for application programs to use in translating names to addresses. In the remainder of this chapter, we refer to the name of a socket as an address.

Use of Sockets

Over the last several years a number of excellent books have been written about socket programming from the user's perspective [Stevens, 1998]. This section includes a brief description of a client and server program communicating over a reliable byte stream in the Internet communication domain. For more detailed information on writing network applications, please see the cited references. The client is described first and the server second.

A program that wants to use a socket creates it with the socket system call:

int sock, domain = AF_INET, type = SOCK_STREAM, protocol = 0;
sock = socket(domain, type, protocol);

The type of socket is selected according to the characteristic properties required by the application. In this example, reliable communication is required, so a stream socket (type = SOCK_STREAM) is selected. The domain parameter specifies the communication domain (or protocol family; see Section 11.4) in which the socket should be created, in this case the IPv4 Internet (domain = AF_INET). The final parameter, the protocol, can be used to indicate a specific communication protocol for use in supporting the socket's operation. Protocols are indicated by well-known (standard) constants specific to each communication domain. When zero is used, the system picks an appropriate protocol. The socket system call returns a file descriptor (a small integer; see Section 6.4) that is then used in all subsequent socket operations.

After a socket has been created, the next step depends on the type of socket being used. Since this example is connection oriented, the sockets require a connection before being used. Creating a connection between two sockets usually requires that each socket have an address bound to it, which is simply a way of identifying each end of the communication.

Applications may explicitly specify a socket's address or may permit the system to assign one. The address to be used with a socket must be given in a socket address structure. The format of addresses can vary among domains; to permit a wide variety of different formats, the system treats addresses as variable-length byte arrays, which are prefixed with a length and a tag that identifies their format. Each domain has its own addressing format, which can always be mapped into the most generic one.

A connection is initiated with a connect system call:

int error;
int sock; /* Previously created by a socket() call. */
struct sockaddr_in rmtaddr; /* Assigned by the program. */
int rmtaddrlen = sizeof (struct sockaddr_in);
error =
     connect(sock, (struct sockaddr *)&rmtaddr, rmtaddrlen);

When the connect call completes, the client has a fully functioning communication endpoint on which it can send and receive data.

A server follows a different path once it has created a socket. It must bind itself to an address and then accept incoming connections from clients. The call to bind an address to a socket is as follows:

int error;
int sock;
struct sockaddr_in addr;
int addrlen = sizeof (struct sockaddr_in);
error =
    bind(sock, (struct sockaddr*)&localaddr, localaddrlen);

where sock is the descriptor created by a previous call to socket.

For several reasons, binding a name to a socket was separated from creating a socket. First, sockets are potentially useful without names. If all sockets had to be named, users would be forced to devise meaningless names without reason. Second, in some communication domains, it may be necessary to supply additional information to the system before binding a name to a socket—for example, the "type of service" required when a socket is used. If a socket's name had to be specified at the time that the socket was created, supplying this information would not be possible without further complicating the interface.

In the server process, the socket must be marked to specify that incoming connections are to be accepted on it:

int error, sock, backlog;
error = listen(sock, backlog);

The backlog parameter in the listen call specifies an upper bound on the number of pending connections that should be queued for acceptance.

Connections are then received, one at a time, with

int newsock, sock;
struct sockaddr_in clientaddr;
int clientaddrlen = sizeof(struct sockaddr_in);
newsock =
    accept(sock, (struct sockaddr *)&clientaddr, clientaddrlen);

The accept call returns a new connected socket, as well as the address of the client by specifying the clientaddr and clientaddrlen parameters. The new socket is the one through which communication can take place. The original socket, sock, is used solely for managing the queue of connection requests in the server.

A variety of calls are available for sending and receiving data; these calls are summarized in Table 11.1. The richest of these interfaces are the sendto and recvfrom calls. Besides scatter-gather operations being possible, an address may be specified or received, optional flags are available, and specially interpreted ancillary data or control information may be passed (see Figure 11.1). Ancillary data may include protocol-specific data, such as addressing or options, and also specially interpreted data, called access rights.

Table 11.1. Sending and receiving data on a socket.

Routine

Connected

Disconnected

Address Info

read

Y

N

N

readv

Y

N

N

write

Y

N

N

writev

Y

N

N

recv

Y

Y

N

send

Y

Y

N

recvmsg

Y

Y

Y

sendmsg

Y

Y

Y

Figure 11.1. Data structures for the sendmsg and recvmsg system calls.

[View full size image]

In addition to these system calls, several other calls are provided to access miscellaneous services. The getsockname call returns the locally bound address of a socket, whereas the getpeername call returns the address of the socket at the remote end of a connection. The shutdown call terminates data transmission or reception at a socket, and two ioctl-style calls—setsockopt and getsockopt—can be used to set and retrieve various parameters that control the operation of a socket or of the underlying network protocols. Sockets are discarded with the normal close system call.

The interface to the interprocess-communication facilities was purposely designed to be orthogonal to the existing standard system interfaces—that is, to the open, read, and write system calls. This decision was made to avoid overloading the familiar interface with undue complexity. In addition, the developers thought that using an interface that was completely independent of the filesystem would improve the portability of software because, for example, pathnames would not be involved. Backward compatibility, for the sake of naive processes, was still deemed important; thus, the familiar read-write interface was augmented to permit access to the new communication facilities wherever that made sense (e.g., when connected stream sockets were used).


11.2. Implementation Structure and Overview

The interprocess-communication facilities are layered on top of the networking facilities, as shown in Figure 11.2. Data flows from the application through the socket layer to the networking layer, and vice versa. State required by the socket level is fully encapsulated within it, whereas any protocol-related state is maintained in auxiliary data structures that are specific to the supporting protocols. Responsibility for storage associated with transmitted data is passed from the socket level to the network level. Consistent adherence to this rule assists in simplifying details of storage management. Within the socket layer, the socket data structure is the focus of all activity. The system-call interface routines manage the actions related to a system call, collecting the system-call parameters (see Section 3.2) and converting user data into the format expected by the second-level routines. Most of the socket abstraction is implemented within the second-level routines. All second-level routines have names with a so prefix, and they directly manipulate socket data structures and manage the synchronization between asynchronous activities; these routines are listed in Table 11.2.

Table 11.2. Socket-layer support routines.

Routine

Function

socreate()

create a new socket

sobind()

bind a name to a socket

solisten()

mark a socket as listening for connection requests

soclose()

close a socket

soabort()

abort connection on a socket

soaccept()

accept a pending connection on a socket

soconnect()

initiate a connection to another socket

soconnect2()

create a connection between two sockets

sodisconnect()

initiate a disconnect on a connected socket

sosend()

send data

soreceive()

receive data

soshutdown()

shut down data transmission or reception

sosetopt()

set the value of a socket option

sogetopt()

get the value of a socket option

Figure 11.2. Interprocess-communication implementation layering. The boxes on the left name the standard layers; the boxes on the right name specific examples of the layers that might be used by an individual socket.

[View full size image]

The remainder of this chapter focuses on the implementation of the socket layer. Section 11.3 discusses how memory is managed at the socket level and below in the networking subsystem; Section 11.4 covers the socket and related data structures; Section 11.5 presents the algorithms for connection setup; Section 11.6 discusses data transfer; and Section 11.7 describes connection shutdown. Throughout this chapter, references to the supporting facilities provided by the network-communication protocols are made with little elaboration. A complete description of the interaction between the network protocols and the socket layer appears in Chapter 12, and the internals of the network protocols are presented in Chapter 13.


11.3. Memory Management

The requirements placed on a memory-management scheme by interprocess-communication and network protocols tend to be substantially different from those of other parts of the operating system. Although all require the efficient allocation and reclamation of memory, communication protocols in particular need memory in widely varying sizes. Memory is needed for variable-sized structures such as communication protocol packets. Protocol implementations must frequently prepend headers or remove headers from packetized data. As packets are sent and received, buffered data may need to be divided into packets, and received packets may be combined into a single record. In addition, packets and other data objects must be queued when awaiting transmission or reception. A special-purpose memory-management facility exists for use by the interprocess-communication and networking systems to address these needs.

Mbufs

The memory-management facilities revolve around a data structure called an mbuf (see Figure 11.3). Mbufs, or memory buffers, vary in size depending on what they contain. All mbufs contain a fixed, m_hdr structure that keeps track of various bits of bookkeeping about the mbuf. An mbuf that contains only data has space for 234 bytes (256 bytes total for the mbuf minus 22 bytes for the mbuf header). All structure sizes are calculated for 32-bit processors.

Figure 11.3. Memory-buffer (mbuf) data structure.

For large messages, the system can associate larger sections of data with an mbuf by referencing an external mbuf cluster from a private virtual memory area. The size of an mbuf cluster may vary by architecture, and is specified by the macro MCLBYTES (which is 2 Kbytes on the PC).

Data are stored either in the internal data area or in an external cluster, but never in both. To access data in either location a data pointer within the mbuf is used. In addition to the data-pointer field, a length field is also maintained. The length field shows the number of bytes of valid data to be found at the data-pointer location. The data and length fields allow routines to trim data efficiently at the start or end of an mbuf. In deletion of data at the start of an mbuf, the pointer is incremented and the length is decremented. To delete data from the end of an mbuf, the length is decremented, but the data pointer is left unchanged. When space is available within an mbuf, data can be added at either end. This flexibility to add and delete space without copying is particularly useful in communication-protocol implementation. Protocols routinely strip protocol information off the front or back of a message before the message's contents are handed to a higher-level processing module, or they add protocol information as a message is passed to lower levels.

Multiple mbufs can be linked to hold an arbitrary quantity of data. This linkage is done with the m_next field of the mbuf. By convention, a chain of mbufs linked in this way is treated as a single object. For example, the communication protocols build packets from chains of mbufs. A second field, m_nextpkt, links objects built from chains of mbufs into lists of objects. Throughout our discussions, a collection of mbufs linked together with the m_next field will be called a chain; chains of mbufs linked together with the m_nextpkt field will be called a queue.

Each mbuf is typed according to its use. This type serves two purposes. The only operational use of the type is to distinguish optional components of a message in an mbuf chain that is queued for reception on a socket data queue. Otherwise, the type information is used in maintaining statistics about storage use and, if there are problems, as an aid in tracking mbufs.

The mbuf flags are logically divided into two sets: flags that describe the usage of an individual mbuf and those that describe an object stored in an mbuf chain. The flags describing an mbuf specify whether the mbuf references external storage (M_EXT), whether the second set of header fields is present (M_PKTHDR), and whether the mbuf completes a record (M_EOR). A packet normally would be stored in an mbuf chain (of one or more mbufs) with the M_PKTHDR flag set on the first mbuf of the chain. The mbuf flags describing the packet would be set in the first mbuf and could include either the broadcast flag (M_BCAST) or the multicast flag (M_MCAST). The latter flags specify that a transmitted packet should be sent as a broadcast or multicast, respectively, or that a received packet was sent in that manner.

If the M_PKTHDR flag is set on an mbuf, the mbuf has a second set of header fields immediately following the standard header. This addition causes the mbuf data area to shrink from 234 bytes to 210 bytes. The packet header, which is shown in Figure 11.4, is only used on the first mbuf of a chain. It includes several fields: a pointer to the interface on which the packet was received, the total length of the packet, a pointer to the packet header, two fields relating to packet checksum calculation, and a pointer to a list of arbitrary tags.

Figure 11.4. Memory-buffer (mbuf) data structure with M_PKTHDR.

An mbuf that uses external storage is marked with the M_EXT flag. Here, a different header area overlays the internal data area of an mbuf. The fields in this header, which is shown in Figure 11.5 (on page 446), describe the external storage, including the start of the buffer and its size. One field is designated to point to a routine to free the buffer, in theory allowing various types of buffers to be mapped by mbufs. In the current implementation, however, the free function is not used, and the external storage is assumed to be a standard mbuf cluster. An mbuf may be both a packet header and have external storage, in which case the standard mbuf header is followed by the packet header and then the external storage header.

Figure 11.5. Memory-buffer (mbuf) data structure with external storage.

The ability to refer to mbuf clusters from an mbuf permits data to be referenced by different entities within the network code without a memory-to-memory copy operation. When multiple copies of a block of data are required, the same mbuf cluster is referenced from multiple mbufs. A single, global array of reference counts is maintained for the mbuf clusters to support this style of sharing (see the next subsection).

Mbufs have fixed-sized, rather than variable-sized, data areas for several reasons. First, the fixed size minimizes memory fragmentation. Second, communication protocols are frequently required to prepend or append headers to existing data areas, split data areas, or trim data from the beginning or end of a data area. The mbuf facilities are designed to handle such changes without reallocation or copying whenever possible. Finally, the dtom() function, described in the subsection on mbuf utility routines later in this section, would be much more expensive if mbufs were not fixed in size.

Since the mbuf is the central object of all of the networking subsystems, it has undergone modification with each large change in the code. It now contains a flags field and two optional sets of header fields. The data pointer replaces a field used as an offset in the initial version of the mbuf. The use of an offset was not portable when the data referenced could be in an mbuf cluster. The addition of a flags field allowed the use of a flag indicating external storage. Earlier versions tested the magnitude of the offset to see whether the data were in the internal mbuf data area. The addition of the broadcast flag allowed network-level protocols to know whether packets were received as link-level broadcasts, as was required for standards conformance. Several other flags have been added for use by specific protocols as well as to handle fragment processing.

The optional header fields have undergone the largest changes since 4.4BSD. The two headers were originally designed to avoid redundant calculations of the size of an object, to make it easier to identify the incoming network interface of a received packet, and to generalize the use of external storage by an mbuf. Since 4.4BSD the packet header has been expanded to include information on checksum calculation (a traditionally expensive operation that can now be done in hardware) and an arbitrary set of tags.

Tags are fixed-size structures that can point to arbitrary pieces of memory and are used to store information relevant to different modules within the networking subsystem. Each tag has a link to the next tag in the list, a 16-bit ID, a 16-bit length, and a 32-bit cookie. The cookie is used to identify which module owns the tag, and the type is a piece of data that is private to the module. Tags are used to carry information about a packet that should not be placed into the packet itself. Examples of these tags are given in Section 13.10.

The design has not been completely successful. It is probably not worth the complexity of having a variable-sized header on an mbuf for the packet header; instead, those fields probably should have been included in all mbufs, even if they were not used. Several of the new packet header fields could be combined because they duplicate functionality; for example, the checksum fields could be implemented using tags.

Storage-Management Algorithms

Providing the system with a network stack capable of symmetric multiprocessing required a complete rework of the memory allocation algorithms underlying the mbuf code. Whereas previous versions of BSD allocated memory with the system allocator and then carved it up for mbufs and clusters, such a simple technique does not work when using multiple CPUs.

FreeBSD 5.2 allocates virtual memory among a series of lists for use by the network memory allocation code. Each CPU has its own private container of mbufs and clusters. There is also a single, general pool of mbufs and clusters from which allocations are attempted when a per-CPU list is empty or to which memory is freed when a per-CPU list is full. A uniprocessor system acts as if it is an SMP system with one CPU, which means that it has one per-CPU list as well as the general one.

At system boot time the memory allocator initially fills up each, per-CPU, list with a tunable number of mbufs and clusters (see Section 14.6). The general list is initialized and left empty; it is used only in cases of high load, when a per-CPU list is starved for memory.

The reference counts for clusters are managed as an array, separately from the clusters themselves. The array is large enough for every mbuf cluster that could be allocated by the system. The memory dedicated to mbufs and clusters is set based on the kernel parameter maxusers, which is itself based on the amount of physical memory in the system. Basing the amount of memory dedicated to the networking subsystem on the amount of physical memory gives a good default value but should be overridden when a system is dedicated to networking tasks such as a Web server, firewall, or router.

Mbuf-allocation requests indicate either that they must be fulfilled immediately or that they can wait for available resources. If a request is marked as "can wait" and the requested resources are unavailable, the process is put to sleep to await available resources. Although a nonblocking allocation request is no longer necessary for code that executes at interrupt level, the networking code still operates as if this is the case. If mbuf allocation has reached its limit or memory is unavailable, the mbuf-allocation routines ask the network-protocol modules to give back any available resources that they can spare. A nonblocking request will fail if no resources are available.

An mbuf-allocation request is made through a call to m_get(), m_gethdr() or through an equivalent macro. An mbuf is retrieved from the appropriate per-CPU list by the mb_alloc() function and is then initialized. For m_gethdr(), the mbuf is initialized with the optional packet header. The MCLGET macro adds an mbuf cluster to an mbuf.

Release of mbuf resources is straightforward: m_free() frees a single mbuf, and m_freem() frees a chain of mbufs. When an mbuf that references an mbuf cluster is freed, the reference count for the cluster is decremented. Mbuf clusters are placed onto the appropriate per-CPU list when their reference counts reach zero.

Mbuf Utility Routines

Many useful utility routines exist for manipulating mbufs within the kernel networking subsystem. Those routines that will be used in Chapter 12 are described briefly here.

The m_copym() routine makes a copy of an mbuf chain starting at a logical offset, in bytes, from the start of the data. This routine may be used to copy all or only part of a chain of mbufs. If an mbuf is associated with an mbuf cluster, the copy will reference the same data by incrementing the reference count on the cluster; otherwise, the data portion is copied as well. The m_copydata() function is similar, but it copies data from an mbuf chain into a caller-provided buffer. This buffer is not an mbuf, or chain, but a plain area of memory pointed to by a traditional C pointer.

The m_adj() routine adjusts the data in an mbuf chain by a specified number of bytes, shaving data off either the front or back. No data are ever copied; m_adj() operates purely by manipulating the offset and length fields in the mbuf structures.

The mtod() macro takes a pointer to an mbuf header and a data type and returns a pointer to the data in the buffer, cast to the given type. The dtom() function is the inverse: It takes a pointer to an arbitrary address in the data of an mbuf and returns a pointer to the mbuf header (rather than to the head of the mbuf chain). This operation is done through simple truncation of the data address to an mbuf-sized boundary. This function works only when data reside within the mbuf.

The m_pullup() routine rearranges an mbuf chain such that a specified number of bytes reside in a contiguous data area within the mbuf (not in external storage). This operation is used so that objects such as protocol headers are contiguous and can be treated as normal data structures, and so that dtom() will work when the object is freed. If there is room, m_pullup() will increase the size of the contiguous region up to the maximum size of a protocol header in an attempt to avoid being called in the future.

The M_PREPEND() macro adjusts an mbuf chain to prepend a specified number of bytes of data. If possible, space is made in place, but an additional mbuf may have to be allocated at the beginning of the chain. It is not currently possible to prepend data within an mbuf cluster because different mbufs might refer to data in different portions of the cluster.


11.4. Data Structures

Sockets are the basic objects used by processes communicating over a network. A socket's type defines the basic set of communication semantics, whereas the communication domain defines auxiliary properties important to the use of the socket and may refine the set of available communication semantics. Table 11.3 shows the four types of sockets currently supported by the system. To create a new socket, applications must specify its type and the communication domain. The request may also indicate a specific network protocol to be used by the socket. If no protocol is specified, the system selects an appropriate protocol from the set of protocols supported by the communication domain. If the communication domain is unable to support the type of socket requested (i.e., no suitable protocol is available), the request will fail.

Table 11.3. Socket types supported by the system.

Name

Type

Properties

SOCK_STREAM

stream

reliable, sequenced, data transfer; may support out-of-band data

SOCK_DGRAM

datagram

unreliable, unsequenced, data transfer, with message boundaries preserved

SOCK_SEQPACKET

sequenced packet

reliable, sequenced, data transfer, with message boundaries preserved

SOCK_RAW

raw

direct access to the underlying communication protocols

Sockets are described by a socket data structure that is dynamically created at the time of a socket system call. Communication domains are described by a domain data structure that is statically defined within the system based on the system's configuration (see Section 14.5). Communication protocols within a domain are described by a protosw structure that is also statically defined within the system for each protocol implementation configured. Having these structures defined statically means that protocol modules cannot be loaded or unloaded at run time.

When a request is made to create a socket, the system uses the value of the communication domain to search linearly the list of configured domains. If the domain is found, the domain's table of supported protocols is consulted for a protocol appropriate for the type of socket being created or for a specific protocol requested. (A wildcard entry may exist for a raw socket.) Should multiple protocol entries satisfy the request, the first is selected. We shall begin discussion of the data structures by examining the domain structure. The protosw structure is discussed in Section 12.1.

Communication Domains

The domain structure is shown in Figure 11.6. The dom_name field is the ASCII name of the communication domain. (In the original design, communication domains were to be specified with ASCII strings; they are now specified with manifest constants.) The dom_family field identifies the address family used by the domain; some possible values are shown in Table 11.4. Address families refer to the addressing structure of a domain. An address family generally has an associated protocol family. Protocol families refer to the suite of communication protocols of a domain used to support the communication semantics of a socket. The dom_protosw field points to the table of functions that implement the protocols supported by the communication domain, and the dom_NPROTOSW pointer marks the end of the table. The remaining entries contain pointers to domain-specific routines used in the management and transfer of access rights and fields relating to routing initialization for the domain.

Table 11.4. Address families.

Name

Description

AF_LOCAL

(AF_UNIX) local communication

AF_INET

Internet (TCP/IP)

AF_INET6

Internet Version 6 (TCP/IPv6)

AF_NS

Xerox Network System (XNS) architecture

AF_ISO

OSI network protocols

AF_CCITT

CCITT protocols, e.g., X.25

AF_SNA

IBM System Network Architecture (SNA)

AF_DLI

direct link interface

AF_LAT

local-area-network terminal interface

AF_APPLETALK

AppleTalk network

AF_ROUTE

communication with kernel routing layer

AF_LINK

raw link-layer access

AF_IPX

Novell Internet protocol

Figure 11.6. Communication-domain data structure.

Sockets

The socket data structure is shown in Figure 11.7 (on page 452). Storage for the socket structure is allocated by the UMA zone allocator (described in Section 5.3). Sockets contain information about their type, the supporting protocol in use, and their state (Table 11.5 on page 453). Data being transmitted or received are queued at the socket as a list of mbuf chains. Various fields are present for managing queues of sockets created during connection establishment. Each socket structure also holds a process-group identifier. The process-group identifier is used in delivering the SIGURG and SIGIO signals. SIGURG is sent when an urgent condition exists for a socket, and SIGIO is used by the asynchronous I/O facility (see Section 6.4). The socket contains an error field, which is needed for storing asynchronous errors to be reported to the owner of the socket.

Table 11.5. Socket states.

State

Description

SS_NOFDREF

no file-table reference

SS_ISCONNECTED

connected to a peer

SS_ISCONNECTING

in process of connecting to peer

SS_ISDISCONNECTING

in process of disconnecting from peer

SS_CANTSENDMORE

cannot send more data to peer

SS_CANTRCVMORE

cannot receive more data from peer

SS_RCVATMARK

at out-of-band mark on input

SS_ISCONFIRMING

peer awaiting connection confirmation

SS_NBIO

nonblocking I/O

SS_ASYNC

asynchronous I/O notification

SS_INCOMP

connection is incomplete and not yet accepted

SS_COMP

connection is complete but not yet accepted

SS_ISDISCONNECTED

socket is disconnected from peer

Figure 11.7. Socket data structure.

[View full size image]

Sockets are located through a process's file descriptor via the file table. When a socket is created, the f_data field of the file structure is set to point at the socket structure, and the f_ops field is set to point to the set of routines defining socket-specific file operations. In this sense, the socket structure is a direct parallel of the vnode structure used by the filesystems.

The socket structure acts as a queueing point for data being transmitted and received. As data enter the system as a result of system calls, such as write or send, the socket layer passes the data to the networking subsystem as a chain of mbufs for immediate transmission. If the supporting protocol module decides to postpone transmission of the data, or if a copy of the data is to be maintained until an acknowledgment is received, the data are queued in the socket's send queue. When the network has consumed the data, it discards them from the outgoing queue. On reception, the network passes data up to the socket layer, also in mbuf chains, where they are then queued until the application makes a system call to request them. The socket layer can also make a callback to an internal kernel client of the network when data arrive, allowing the data to be processed without a context switch. Callbacks are used by the NFS server (see Chapter 9).

To avoid resource exhaustion, sockets impose upper bounds on the number of bytes of data that can be queued in a socket data buffer and also on the amount of storage space that can be used for data. This high watermark is initially set by the protocol, although an application can change the value up to a system maximum, normally 256 Kbyte. The network protocols can examine the high watermark and use the value in flow-control policies. A low watermark also is present in each socket data buffer. The low watermark allows applications to control data flow by specifying a minimum number of bytes required to satisfy a reception request, with a default of 1 byte and a maximum of the high watermark. For output, the low watermark sets the minimum amount of space available before transmission can be attempted; the default is the size of an mbuf cluster. These values also control the operation of the select system call when it is used to test for ability to read or write the socket.

When connection indications are received at the communication-protocol level, the connection may require further processing to complete. Depending on the protocol, that processing may be done before the connection is returned to the listening process, or the listening process may be allowed to confirm or reject the connection request. Sockets used to accept incoming connection requests maintain two queues of sockets associated with connection requests. The list of sockets headed by the so_incomp field represents a queue of connections that must be completed at the protocol level before being returned. The so_comp field heads a list of sockets that are ready to be returned to the listening process. Like the data queues, the queues of connections also have an application-controllable limit. The limit applies to both queues. Because the limit may include sockets that cannot yet be accepted, the system enforces a limit 50 percent larger than the nominal limit.

Although a connection may be established by the network protocol, the application may choose not to accept the established connection or may close down the connection immediately after discovering the identity of the client. It is also possible for a network protocol to delay completion of a connection until after the application has obtained control with the accept system call. The application might then accept or reject the connection explicitly with a protocol-specific mechanism. Otherwise, if the application does a data transfer, the connection is confirmed; if the application closes the socket immediately, the connection is rejected.

Socket Addresses

Sockets may be labeled so that peers can connect to them. The socket layer treats an address as an opaque object. Applications supply and receive addresses as tagged, variable-length byte strings. Addresses are placed in mbufs within the socket layer. A structure called a sockaddr, shown in Figure 11.8, is used as a template for referring to the identifying tag and length of each address. Most protocol layers support a single address type as identified by the tag, known as the address family.

Figure 11.8. Socket-address template structure.

It is common for addresses passed in by an application to reside in mbufs only long enough for the socket layer to pass them to the supporting protocol for transfer into a fixed-sized address structure—for example, when a protocol records an address in a protocol control block. The sockaddr structure is the common means by which the socket layer and network-support facilities exchange addresses. The size of the generic data array was chosen to be large enough to hold many types of addresses directly, although generic code cannot depend on having sufficient space in a sockaddr structure for an arbitrary address. The local communication domain (formerly known as the UNIX domain), for example, stores filesystem pathnames in mbufs and allows socket names as large as 104 bytes, as shown in Figure 11.9. Both IPv4 and IPv6 use a fixed-size structure that combines an Internet address and a port number. The difference is in the size of the address (4 bytes for IPv4 and 16 bytes for IPv6) and in the fact that IPv6 address structures carry other information (the scope and flow info). Both Internet protocols reserve space for addresses in a protocol-specific control-block data structure and free up mbufs that contain addresses after copying the addresses.

Figure 11.9. Local-domain, IPv4 and IPv6 address structures.

[View full size image]

Locks

Section 4.3 discussed the need for locking structures in an SMP kernel. The networking subsystem uses these locks internally to protect its data structures.

When SMP features were first introduced, the entire networking subsystem was placed, with the rest of the kernel, under the giant lock. During the development of FreeBSD 5.2 several pieces of networking code were modified to run without the giant lock, but at this time parts of the system still use it. Whenever a user process calls the networking subsystem (i.e., they create a socket, make a connection, send or receive data), they acquire the giant lock. Other locks are acquired and released when interacting with lower layers of the protocols and network devices. Specific instances of networking subsystem locks are discussed in the section in which they are most relevant.


11.5. Connection Setup

For two processes to pass information between them, an association must be established. The steps involved in creating an association (socket, connect, listen, accept, etc.) were described in Section 11.1. In this section, we study the operation of the socket layer in establishing associations. Since the state associated with a connectionless transfer of data is fully encapsulated in each message that is sent, our discussion will focus on connection-based associations established with the connect, listen, and accept system calls.

Connection establishment in the client-server model is asymmetric. A client actively initiates a connection to obtain service, whereas a server passively accepts connections to provide service. Figure 11.10 (on page 456) shows the state-transition diagram for a socket used to initiate or accept connections. State transitions are initiated either by user actions (i.e., system calls) or by protocol actions that result from receiving network messages or servicing timers that expire.

Figure 11.10. Socket state transitions during process rendezvous.

[View full size image]

Sockets are normally used to send and receive data. When they are used in establishing a connection, they are treated somewhat differently. If a socket is to be used to accept a connection, a listen system call must be used. The listen call invokes solisten(), which notifies the supporting protocol that the socket will be receiving connections, establishes an empty list of pending connections at the socket (through the so_comp field), and then marks the socket as accepting connections, SO_ACCEPTCON. At the time a listen is done, a backlog parameter is specified by the application. This parameter sets a limit on the number of incoming connections that the system will queue awaiting acceptance by the application. (The system enforces a maximum on this limit.) Once a socket is set up to receive connections, the remainder of the work in creating connections is managed by the protocol layers. For each connection established at the server side, a new socket is created with the sonewconn() routine. These new sockets may be placed on the socket's queue of partially established connections (see Figure 11.11) while the connections are being completed, or they may be placed directly into the queue of connections ready to be passed to the application via the accept call. The new sockets might be ready to be passed to the application either because no further protocol action is necessary to establish the connection or because the protocol allows the listening process to confirm or reject the connection request. In the latter case, the socket is marked as confirming (state bit SS_CONFlRMlNG) so that the pending connection request will be confirmed or rejected as needed. Once sockets on the queue of partly established connections are ready, they are moved to the queue of connections completed and pending acceptance by an application. When an accept system call is made to obtain a connection, the system verifies that a connection is present on the socket's queue of ready connections. If no connection is ready to be returned, the system puts the process to sleep until one arrives (unless nonblocking I/O is being used with the socket, in which case an error is returned). When a connection is available, the associated socket is removed from the queue, a new file descriptor is allocated to reference the socket, and the result is returned to the caller. If the accept call indicates that the peer's identity is to be returned, the peer's address is obtained from the protocol layer and is copied into the supplied buffer.

Figure 11.11. Connections queued at a socket awaiting an accept call.

[View full size image]

On the client side, an application requests a connection with the connect system call, supplying the address of the peer socket to which to connect. The system verifies that a connection attempt is not already in progress for that socket and then invokes soconnect() to initiate the connection. The soconnect() routine first checks the socket to see whether the latter is already connected. If the socket is already connected, the existing connection is first terminated (this disconnection is done with datagram sockets only). With the socket in an unconnected state, soconnect() makes a request to the protocol layer to initiate the new connection. Once the connection request has been passed to the protocol layer, if the connection request is incomplete, the system puts the process to sleep to await notification by the protocol layer that a completed connection exists. A nonblocking connect may return at this point, but a process awaiting a completed connection will awaken only when the connection request has been completed—either successfully or with an error condition.

A socket's state during connection establishment is managed jointly by the socket layer and the supporting protocol layer. The socket's state value is never altered directly by a protocol; to promote modularity, all modifications are performed by surrogate socket-layer routines, such as soisconnected(). These routines modify the socket state as indicated and notify any waiting processes. The supporting protocol layers never use synchronization or signaling facilities directly. Errors that are detected asynchronously are communicated to a socket in its so_error field. For example, if a connection request fails because the protocol layer detects that the requested service is unavailable, the so_error field is set to ECONNREFUSED before the requesting process is awakened. The socket layer always inspects the value of so_error on return from a call to sleep(); this field is used to report errors detected asynchronously by the protocol layers.


11.6. Data Transfer

Most of the work done by the socket layer lies in sending and receiving data. Note that the socket layer itself explicitly refrains from imposing any structure on data transmitted or received via sockets other than optional record boundaries.

Within the overall interprocess-communication model, any data interpretation or structuring is logically isolated in the implementation of the communication domain. An example of this logical isolation is the ability to pass file descriptors between processes using local-domain sockets.

Sending and receiving data can be done with any one of several system calls. The system calls vary according to the amount of information to be transmitted and received, and according to the state of the socket doing the operation. For example, the write system call may be used with a socket that is in a connected state, since the destination of the data is implicitly specified by the connection. The sendto or sendmsg system calls, however, allow the process to specify the destination for a message explicitly. Likewise, when data are received, the read system call allows a process to receive data on a connected socket without receiving the sender's address; the recvfrom and recvmsg system calls allow the process to retrieve the incoming message and the sender's address. The differences between these calls were summarized in Section 11.1. The recvmsg and sendmsg system calls allow scatter-gather I/O with multiple user-provided buffers. In addition, recvmsg reports additional information about a received message, such as whether it was expedited (out of band), whether it completes a record, or whether it was truncated because a buffer was too small. The decision to provide many different system calls rather than only a single general interface is debatable. It would have been possible to implement a single system-call interface and to provide simplified interfaces to applications via user-level library routines. However, the single system call would have to be the most general call, which has somewhat higher overhead. Internally, all transmission and reception requests are converted to a uniform format and are passed to the socket-layer sendit() and recvit() routines, respectively.

Transmitting Data

The sendit() routine is responsible for gathering all system-call parameters that the application has specified into the kernel's address space (except the actual data) and then for invoking the sosend() routine to do the transmission. The parameters may include the following components, illustrated in Figure 11.1:

  • An address to which data will be sent, if the socket has not been connected

  • Optional ancillary data (control data) associated with the message; ancillary data can include protocol-specific data associated with a message, protocol option information, or access rights

  • Normal data, specified as an array of buffers (see Section 6.4)

  • Optional flags, including out-of-band and end-of-record flags

The sosend() routine handles most of the socket-level data-transmission options, including requests for transmission of out-of-band data and for transmission without network routing. This routine is also responsible for checking socket state— for example, seeing whether a required connection has been made, whether transmission is still possible on the socket, and whether a pending error should be reported rather than transmission attempted. In addition, sosend() is responsible for putting processes to sleep when their data transmissions exceed the buffering available in the socket's send buffer. The actual transmission of data is done by the supporting communication protocol; sosend() copies data from the user's address space into mbufs in the kernel's address space and then makes calls to the protocol to transfer the data.

Most of the work done by sosend() lies in checking the socket state, handling flow control, checking for termination conditions, and breaking up an application's transmission request into one or more protocol transmission requests. The request must be broken up only when the size of the user's request plus the number of data queued in the socket's send data buffer exceeds the socket's high watermark. It is not permissible to break up a request if the protocol is atomic, because each request made by the socket layer to the protocol modules implicitly indicates a boundary in the data stream. Most datagram protocols are of this type. Honoring each socket's high watermark ensures that a protocol will always have space in the socket's send buffer to enqueue unacknowledged data. It also ensures that no process, or group of processes, can monopolize system resources.

For sockets that guarantee reliable data delivery, a protocol will normally maintain a copy of all transmitted data in the socket's send queue until receipt is acknowledged by the receiver. Protocols that provide no assurance of delivery normally accept data from sosend() and directly transmit the data to the destination without keeping a copy. But sosend() itself does not distinguish between reliable and unreliable delivery.

Sosend() always ensures that a socket's send buffer has enough space available to store the next section of data to be transmitted. If a socket has insufficient space in its send buffer to hold all the data to be transmitted, sosend() uses the following strategy. If the protocol is atomic, sosend() verifies that the message is no larger than the send buffer size; if the message is larger, it returns an EMSGSIZE error. If the available space in the send queue is less than the send low watermark, the transmission is deferred. If the process is not using nonblocking I/O, the process is put to sleep until more space is available in the send buffer. Otherwise, an error is returned. When space is available, a protocol transmit request is formulated according to the available space in the send buffer. Sosend() copies data from the user's address space into mbuf clusters whenever the data are larger than the minimum cluster size (specified by MINCLSIZE). If a transmission request for a nonatomic protocol is large, each protocol transmit request will normally contain a full mbuf cluster. Although additional data could be appended to the mbuf chain before delivery to the protocol, it is preferable to pass the data to lower levels immediately. This strategy allows better pipelining, because data reach the bottom of the protocol stack earlier and can begin physical transmission sooner. This procedure is repeated until insufficient space remains; it resumes each time additional space becomes available.

This strategy tends to preserve the application-specified message size and helps to avoid fragmentation at the network level. The latter benefit is important because system performance is significantly improved when data-transmission units are large—for example, the mbuf cluster size.

When the receiver or network is slower than the transmitter, the underlying connection-based transmission protocols usually apply some form of flow control to delay the sender's transmission. In this case, the amount of data that the receiver will allow the sender to transmit can decrease to the point that the sender's natural transmission size drops below its optimal value. To retard this effect, sosend() delays transmission rather than breaking up the data to be transmitted in the hope that the receiver will reopen its flow control window and allow the sender to perform optimally. The effect of this scheme is fairly subtle and is also related to the networking subsystem's optimized handling of incoming data packets that are a multiple of the machine's page size (described in Section 12.8).

The sosend() routine, in manipulating a socket's send data buffer, takes care to ensure that access to the buffer is synchronized among multiple sending processes. It does so by bracketing accesses to the data structure with calls to sblock() and sbunlock().

Receiving Data

The soreceive() routine receives data queued at a socket. As the counterpart to sosend(), soreceive() appears at the same level in the internal software structure and does similar tasks. Three types of data may be queued for reception at a socket: in-band data, out-of-band data, and ancillary data, such as access rights. In-band data may also be tagged with the sender's address. Handling of out-of-band data varies by protocol. They may be placed at the beginning of the receive buffer or at the end of the buffer to appear in order with other data, or they may be managed in the protocol layer separate from the socket's receive buffer. In the first two cases, they are returned by normal receive operations. In the final case, they are retrieved through a special interface when requested by the user. These options allow varying styles of urgent data transmission.

Soreceive() checks the socket's state, including the received data buffer, for incoming data, errors, or state transitions, and processes queued data according to their type and the actions specified by the caller. A system-call request may specify that only out-of-band data should be retrieved (MSG_OOB) or that data should be returned but not removed from the data buffer (by specifying the MSG_PEEK flag). Receive calls normally return as soon as the low watermark is reached. Because the default is one octet, the call returns when any data are present. The MSG_WAITALL flag specifies that the call should block until it can return all the requested data, if possible. Alternatively, the MSG_DONTWAIT flag causes the call to act as though the socket was in nonblocking mode, returning EAGAIN rather than blocking.

Data present in the receive data buffer are organized in one of several ways, depending on whether message boundaries are preserved. There are three common cases for stream, datagram, and sequenced-packet sockets. In the general case, the receive data buffer is organized as a list of messages (see Figure 11.12). Each message can include a sender's address (for datagram protocols), ancillary data, and normal data. Depending on the protocol, it is also possible for expedited or out-of-band data to be placed into the normal receive buffer. Each mbuf chain on a list represents a single message or, for the final chain, a possibly incomplete record. Protocols that supply the sender's address with each message place a single mbuf containing the address at the front of message. Immediately following any address is an optional mbuf containing any ancillary data. Regular data mbufs follow the ancillary data. Names and ancillary data are distinguished by the type field in an mbuf; addresses are marked as MT_SONAME, whereas ancillary data are tagged as MT_CONTROL. Each message other than the final one is considered to be terminated. The final message is terminated implicitly when an atomic protocol is used, such as most datagram protocols. Sequenced packet protocols could treat each message as an atomic record, or they could support records that could be arbitrarily long (as is done in OSI). In the latter case, the final record in the buffer might or might not be complete, and a flag on the final mbuf, M_EOR, marks the termination of a record. Record boundaries (if any) are generally ignored by a stream protocol. However, transition from out-of-band data to normal data in the buffer, or presence of ancillary data, causes logical boundaries. A single receive operation never returns data that cross a logical boundary. Note that the storage scheme used by sockets allows them to compact data of the same type into the minimal number of mbufs required to hold those data.

Figure 11.12. Data queueing for datagram socket.

On entry to soreceive(), a check is made to see whether out-of-band data are being requested. If they are, the protocol layer is queried to see whether any such data are available; if the data are available, they are returned to the caller. Since regular data cannot be retrieved simultaneously with out-of-band data, soreceive() then returns. Otherwise, data from the normal queue have been requested. The soreceive() function first checks whether the socket is in confirming state, with the peer awaiting confirmation of a connection request. If it is, no data can arrive until the connection is confirmed, and the protocol layer is notified that the connection should be completed. Soreceive() then checks the receive-data-buffer character count to see whether data are available. If they are, the call returns with at least the data currently available. If no data are present, soreceive() consults the socket's state to find out whether data might be forthcoming. Data may no longer be received because the socket is disconnected (and a connection is required to receive data) or because the reception of data has been terminated with a shutdown by the socket's peer. In addition, if an error from a previous operation was detected asynchronously, the error needs to be returned to the user; soreceive() checks the so_error field after checking for data. If no data or error exists, data might still arrive, and if the socket is not marked for nonblocking I/O, soreceive() puts the process to sleep to await the arrival of new data.

When data arrive for a socket, the supporting protocol notifies the socket layer by calling sorwakeup(). Soreceive() can then process the contents of the receive buffer, observing the data-structuring rules described previously. Soreceive() first removes any address that must be present, then optional ancillary data, and finally normal data. If the application has provided a buffer for the receipt of ancillary data, they are passed to the application in that buffer; otherwise, they are discarded. The removal of data is slightly complicated by the interaction between in-band and out-of-band data managed by the protocol. The location of the next out-of-band datum can be marked in the in-band data stream and used as a record boundary during in-band data processing. That is, when an indication of out-of-band data is received by a protocol that holds out-of-band data separately from the normal buffer, the corresponding point in the in-band data stream is marked. Then, when a request is made to receive in-band data, only data up to the mark will be returned. This mark allows applications to synchronize the in-band and out-of-band data streams so that, for example, received data can be flushed up to the point at which out-of-band data are received. Each socket has a field, so_oobmark, that contains the character offset from the front of the receive data buffer to the point in the data stream at which the last out-of-band message was received. When in-band data are removed from the receive buffer, the offset is updated so that data past the mark will not be mixed with data preceding the mark. The SS_RCVATMARK bit in a socket's state field is set when so_oobmark reaches zero to show that the out-of-band data mark is at the beginning of the socket receive buffer. An application can test the state of this bit with the SIOCATMARK ioctl call to find out whether all in-band data have been read up to the point of the mark.

Once data have been removed from a socket's receive buffer, soreceive() updates the state of the socket and notifies the protocol layer that data have been received by the user. The protocol layer can use this information to release internal resources, to trigger end-to-end acknowledgment of data reception, to update flow-control information, or to start a new data transfer. Finally, if any access rights were received as ancillary data, soreceive() passes them to a communication-domain-specific routine to convert them from their internal representation to the external representation.

The soreceive() function returns a set of flags that are supplied to the caller of the recvmsg system call via the msg_flags field of the msghdr structure (see Figure 11.1). The possible flags include MSG_EOR to specify that the received data complete a record for a nonatomic sequenced packet protocol, MSG_OOB to specify that expedited (out-of-band) data were received from the normal socket receive buffer, MSG_TRUNC to specify that an atomic record was truncated because the supplied buffer was too small, and MSG_CTRUNC to specify that ancillary data were truncated because the control buffer was too small.


11.7. Socket Shutdown

Although closing a socket and reclaiming its resources at first glance appears to be a straightforward operation, it can be complicated. The complexity arises because of the implicit semantics of the close system call. In certain situations (e.g., when a process exits), a close call is never expected to fail. However, when a socket promising reliable delivery of data is closed with data still queued for transmission or awaiting acknowledgment of reception, the socket must attempt to transmit the data, perhaps indefinitely, for the close call to maintain the socket's advertised semantics. If the socket discards the queued data to allow the close to complete successfully, it violates its promise to deliver data reliably. Discarding data can cause naive processes, which depend on the implicit semantics of close, to work unreliably in a network environment. However, if sockets block until all data have been transmitted successfully, then, in some communication domains, a close may never complete!

The socket layer compromises in an effort to address this problem yet maintain the semantics of the close system call. Figure 11.13 (on page 464) shows the possible state transitions for a socket from a connected to a closed state. In normal operation, closing a socket causes any queued but unaccepted connections to be discarded. If the socket is in a connected state, a disconnect is initiated. The socket is marked to indicate that a file descriptor is no longer referencing it, and the close operation returns successfully. When the disconnect request completes, the network support notifies the socket layer, and the socket resources are reclaimed. The network layer may attempt to transmit any data queued in the socket's send buffer, although there is no guarantee that it will. However, commonly used connection-oriented protocols generally attempt to transmit any queued data asynchronously after the close call returns, preserving the normal semantics of close on a file.

Figure 11.13. Socket-state transitions during shutdown.

[View full size image]

Alternatively, a socket may be marked explicitly to force the application process to linger when closing until pending data have drained and the connection has shut down. This option is marked in the socket data structure using the setsockopt system call with the SO_LINGER option. When an application indicates that a socket is to linger, it also specifies a duration for the lingering period. The application can then block for as long as the specified duration while waiting for pending data to drain. If the lingering period expires before the disconnect is completed, the socket layer then notifies the network that it is closing, possibly discarding any data still pending. Some protocols handle the linger option differently. In particular, if the linger option is set with a duration of zero, the protocol may discard pending data rather than attempt to deliver them asynchronously.


11.8. Local Interprocess-Communication

The socket interfaces are not the only APIs that provide interprocess communication. Applications that wish to divide up work on a single host use semaphores, messages queues, and shared memory to communicate between their processes. Each type of local IPC has different performance characteristics and provides a different form of communication. The local IPC mechanisms currently supported in FreeBSD 5.2 are derived from System V, as described in Bach [1986]. For this reason they are often referred to as System V semaphores, mutexes, and shared memory.

Every type of IPC must make it possible for independently executing processes to rendezvous and find the resources they are sharing. This piece of information must be known to all of them and must be unique enough that no other process could come across the same information by accident. In the local IPC realm this piece of information is called a key.

A key is a long integer that is treated by the cooperating processes as an opaque piece of data, meaning that they do not attempt to decipher or attribute any meaning to it. The library routine, ftok(), is used to generate a key from a pathname. As long as each process uses the same pathname they are guaranteed to get the same key.

All of the local IPC subsystems were designed and implemented to be used in a similar way. Once a process has a key, it uses it to create or retrieve the relevant object, using a subsystem specific get call, which is similar to a file open or creat. To create an object the IPC_CREAT flag is passed as an argument to the get call. All get calls return an integer to be used in all subsequent local IPC system calls. Just like a file descriptor this integer is used to identify the object that the process is manipulating.

Each subsystem has its own way of operating on the underlying object, and these functions are described in the following sections. All control operations, such as retrieving statistics or removing a previously created object, are handled by a subsystem-specific ctl routine. A summary of all the user level APIs is given in Table 11.6, and an excellent introduction to using them can be found in Stevens [1999].

Table 11.6. Local IPC, user level APIs.

Subsystem

Create

Control

Communicate

semaphores

semget

semctl

semop

message queues

msgget

mesgctl

msgrcv, msgsnd

shared memory

shmget

shmctl, shmdt

n/a

Semaphores

A semaphore is the smallest atom of IPC available to a set of cooperating processes. Each semaphore contains a short integer that can be increased or decreased. A process that attempts to reduce the value of the semaphore below 0 will either be blocked or, if called in a nonblocking mode, will return immediately with an errno value of EAGAIN. The concept of semaphores and how they are used in multiprocess programs was originally proposed in Dijkstra & Genuys [1968].

Unlike the semaphores described in most computer science textbooks, semaphores in FreeBSD 5.2 are grouped into arrays so that the code in the kernel can protect the process using them from causing a deadlock. Deadlocks were discussed in terms of locking within the kernel in Section 4.3 but are discussed here are well.

With System V semaphores the deadlock occurs between two user-level processes rather than between kernel threads. A deadlock occurs between two processes, A and B, when they both attempt to take two semaphores, S1 and S2. If process A acquires S1 and process B acquires S2, then a deadlock occurs when process A tries to acquire S2 and process B tries to acquire Sl because there is no way for either process to give up the semaphore that the other one needs to make progress. It is always important when using semaphores that all cooperating processes acquire and release them in the same order to avoid this situation.

The implementation of semaphores in System V protected against deadlock by forcing the user of the API to group their semaphores into arrays and to perform semaphore operations as a sequence of events on the array. If the sequence submitted in the call could cause a deadlock, then an error was returned. The section on semaphores in Bach [1986] points out that this complexity should never have been placed in the kernel, but in order to adhere to the previously defined API, the same complexity exists in FreeBSD as well. At some point in the future the kernel should provide a simpler form of semaphores to replace the current implementation.

Creating and attaching to a semaphore is done with the semget system call. Although semaphores were designed to look like file descriptors, they are not stored in the file descriptor table. All the semaphores in the system are contained in a single table in the kernel, whose size and shape are described by several tunable parameters. This table is protected by the giant lock (see Section 4.3) creating entries in it. This lock is only taken when creating or attaching to a semaphore and is not a bottleneck in the actual use of existing semaphores.

Once a process has created a semaphore, or attached to a preexisting one, it calls semop to perform operations on it. The operations on the semaphore are passed to the system call as an array. Each element of the array includes the semaphore number to operate on (the index into the array returned by the previous semget call), the operation to perform, and a set of flags. The operation is a misnomer because it is not a command but simply a number. If the number is positive, then the corresponding semaphore's value is increased by that amount. If the operation is 0 and the semaphore's value is not 0, then either the process is put to sleep until the value is 0 or, if the IPC_NOWAIT flag was passed, EAGAIN is returned to the caller. When the operation is negative, there are several possible outcomes. If the value of the semaphore was greater than the absolute value of the operation, then the value of the operation is subtracted from the semaphore and the call returns. If subtracting the absolute value of the operation from the semaphore would force its value to be less than zero, then the process is put to sleep, unless the IPC_NOWAIT flag was passed. In that case EAGAIN is returned to the caller.

All of this logic is implemented in the semop system call. The call first does some rudimentary checks to make sure that it has a chance of succeeding, including making sure there is enough memory to execute all the operations in one pass and that the calling process has the proper permissions to access the semaphore. Each semaphore ID returned to a process by the kernel has its own mutex to protect against multiple processes modifying the same semaphore at the same time. The routine locks this mutex and then attempts to perform all the operations passed to it in the array. It walks the array and attempts to perform each operation in order. There is the potential for this call to sleep before it completes all its work. If this situation occurs, then the code rolls back all its work before it goes to sleep. When it reawakens, the routine starts at the beginning of the array and attempts to do the operations again. Either the routine will complete all its work, return with an appropriate error, or go back to sleep. Rolling back all the work is necessary to guarantee the idempotence of the routine. Either all the work is done or none of it is.

Message Queues

A message queue facilitates the sending and receiving of typed, arbitrary-length messages. The sending process adds messages at one end of the queue, and the receiving process removes messages from the other. The queue's size and other characteristics are controlled by a set of tunable kernel parameters. Message queues are inherently half duplex, meaning that one process is always the sender and the other is the receiver, but there are ways to use them as a form of full duplex communication, as we will see later.

The messages passed between the endpoints contain a type and a data area, as shown in Figure 11.14. This data structure should not be confused with the mbufs that are used by the networking code (see Section 11.3). MSGMNB is a tunable kernel parameter that defines the size of a message queue, and therefore the largest possible message that can be sent between two processes, and is set to 2048 by default.

Figure 11.14. Message data structure.

Message queues can be used to implement either a pure first-in first-out queue, where all messages are delivered in the order in which they were sent, or a priority queue, where messages with a certain type can be retrieved ahead of others. This ability is provided by the type field of the message structure.

When a process sends a message, it invokes the msgsnd system call, which checks all the arguments in the call for correctness and then attempts to get enough resources to place the message into the queue. If there aren't enough resources, and the caller did not pass the IPC_NOWAIT flag, then the caller is put to sleep until such time as resources are available. The resources come from a pool of memory that is allocated by the kernel at boot time. The pool is arranged in fixed segments whose length is defined by MSGSSZ. The memory pool is managed as a large array so the segments can be located efficiently.

The kernel data structures that control the message queues in the system are protected by a single lock (msq_mtx), which is taken and held by both msgsnd and msgrcv for the duration of their execution. The use of a single lock for both routines protects the queue from being read and written simultaneously, possibly causing data corruption. It is also a performance bottleneck because it means that all other message queues are blocked when any one of them is being used.

Once the kernel has enough resources, it copies the message into the segments in the array and updates the rest of the data structures related to this queue.

To retrieve a message from the queue, a process calls msgrcv. If the processes are using the queue as a simple fifo, then the receiver passes a 0 in the msgtype argument to this call to retrieve the first available message in the queue. To retrieve the first message in the queue of a particular type, a positive integer is passed. Processes implement a priority queue by using the type as the priority of the message. To implement a full duplex channel, each process picks a different type—say, 1 and 2. Messages of type 1 are from process A, and messages of type 2 are from B. Process A sends messages with type 1 and receives messages with type 2, while process B does exactly the opposite.

After acquiring the message queue mutex, the receive routine finds the correct queue from which to retrieve data, and if there is an appropriate message, it returns data from the segments to the caller. If no data are available and the caller specified the IPC_NOWAIT flag, then the call returns immediately; otherwise, the calling process is put to sleep until there are data to be returned. When a message is retrieved from a message queue, its data are deallocated after they have been delivered to the receiving process.

Shared Memory

Shared memory is used when two or more processes need to communicate large amounts of data between them. Each process stores data in the shared memory just as it would within its own, per-process, memory. Care must be taken to serialize access to the shared memory so that processes do not write over each other. Hence, shared memory is often used with semaphores to synchronize read and write access.

Processes that are using shared memory are really sharing virtual memory (see Chapter 5). When a process creates a segment of shared memory by calling the shmget system call, the kernel allocates a set of virtual memory pages and places a pointer to them in the shared memory handle that is then returned to the calling process. To actually use the shared memory within a process, it must call the shmat system call, which attaches the virtual memory pages into the calling process. The attach routine uses the shared memory handle passed to it as an argument to find the relevant pages and returns an appropriate virtual address to the caller. Once this call completes, the process can then access the memory pointed to by the returned address as it would any other kind of memory.

When the process is through using the shared memory, it detaches from it using the shmdt system call. This routine does not free the associated memory, because other processes may be using it, but it removes the virtual memory mapping from the calling process.

The shared memory subsystem depends on the virtual memory system to do most of the real work (mapping pages, handling dirty pages, etc.), so its implementation is relatively simple.


Exercises

11.1 What limitation in the use of pipes inspired the developers to design alternative interprocess-communication facilities?

11.2 Why are the FreeBSD interprocess-communication facilities designed to be independent of the filesystem for naming sockets?

11.3 Why is interprocess communication layered on top of networking in FreeBSD, rather than the other way around?

11.4 Would a screen editor be considered a naive or a sophisticated program, according to the definitions given in this chapter? Explain your answer.

11.5 What are out-of-band data? What types of socket support the communication of out-of-band data? Describe one use for out-of-band data.

11.6 Give two requirements that interprocess communication places on a memory-management facility.

11.7 How many mbufs and mbuf clusters would be needed to hold a 3024-byte message? Draw a picture of the necessary mbuf chain and any associated mbuf clusters.

11.8 Why does an mbuf have two link pointers? For what is each pointer used?

11.9 Each socket's send and receive data buffers have high and low watermarks. For what are these watermarks used?

11.10 Consider a socket with a network connection that is queued at the socket awaiting an accept system call. Is this socket on the queue headed by the so_comp or by the so_incomp field in the socket structure? What is the use of the queue that does not contain the socket?

11.11 Describe two types of protocols that would immediately place incoming connection requests into the queue headed by the so_comp field in the socket structure.

11.12 How does the protocol layer communicate an asynchronous error to the socket layer?

11.13 Sockets explicitly refrain from interpreting the data that they send and receive. Do you believe that this approach is correct? Explain your answer.

11.14 Why does the sosend() routine ensure there is enough space in a socket's send buffer before making a call to the protocol layer to transmit data?

11.15 How is the type information in each mbuf used in the queueing of data at a datagram socket? How is this information used in the queueing of data at a stream socket?

11.16 Why does the soreceive() routine optionally notify the protocol layer when data are removed from a socket's receive buffer?

11.17 What might cause a connection to linger forever when closing?

11.18 Describe a deadlock between two processes, A and B, that are sharing two semaphores, S1 and S2.

11.19 How can a message queue be used to implement a priority queue? How can it be used to allow full duplex communication?

11.20 Why doesn't the shmdt system call free the underlying shared memory?

*11.21 What effect might storage compaction have on the performance of network-communication protocols?

**11.22 Why is releasing mbuf-cluster storage back to the system complicated? Explain why it might be desirable.

**11.23 In the original design of the interprocess-communication facilities, a reference to a communication domain was obtained with a domain system call,

int d; d = domain("inet");

(where d is a descriptor, much like a file descriptor), and sockets then were created with

s = socket(type, d, protocol);
int s, type, protocol;

What advantages and disadvantages does this scheme have compared to the one that is used in FreeBSD? What effect does the introduction of a domain descriptor type have on the management and use of descriptors within the kernel?

**11.24 Design and implement a simple replacement for Local IPC semaphores that operates on a single semaphore instead of an array. The new system should adhere to the original API to the extent that it should implement a semget, semctl, and semop routine.


References

Bach, 1986. M. J. Bach, The Design of the UNIX Operating System, Prentice-Hall, Englewood Cliffs, NJ, 1986.

Cerf, 1978. V Cerf, "The Catenet Model for Internetworking," Technical Report IEN 48, SRI Network Information Center, Menlo Park, CA, July 1978.

Cohen, 1977. D. Cohen, "Network Control Protocol (NCP) Software," University of Illinois Software Distribution, University of Illinois, Champaign-Urbana, IL, 1977.

Dijkstra & Genuys, 1968. E. Dijkstra & F. Genuys, editor, "Cooperating Sequential Processes," in Programming Languages, pp. 43-112, Academic Press, New York, NY, 1968.

Gurwitz, 1981. R. F. Gurwitz, "VAX-UNIX Networking Support Project—Implementation Description," Technical Report IEN 168, SRI Network Information Center, Menlo Park, CA, January 1981.

Stevens, 1998. R. Stevens, Unix Network Programming Volume 1, Second Edition, Prentice-Hall, Englewood Cliffs, NJ, 1998.

Stevens, 1999. R. Stevens, Unix Network Programming Volume 2, Second Edition, Prentice-Hall, Englewood Cliffs, NJ, 1999.


Chapter 12. Network Communication

In this chapter, we study the internal structure of the network subsystem provided in FreeBSD. The networking facilities provide a framework within which many network architectures may coexist. A network architecture comprises a set of network-communication protocols, the protocol family; conventions for naming communication endpoints, the address family; and any additional facilities that may fall outside the realm of connection management and data transfer. Networking facilities are accessed through the socket abstraction described in Chapter 11. The network subsystem provides a general-purpose framework within which network services are implemented. These facilities include the following:

  • A structured interface to the socket level that allows the development of network-independent application software

  • A consistent interface to the hardware devices used to transmit and receive data

  • Network-independent support for message routing

  • Memory management

We describe the internal structure of the network subsystem in Section 12.1. Then we discuss the interface between the socket layer and the network facilities and examine the interfaces between the layers of software that make up the network subsystem. In Section 12.5, we discuss the routing services used by the network protocols; in Section 12.6, we describe the mechanisms provided to manage buffering and to control congestion. We present the raw-socket interface that provides direct access to lower-level network protocols in Section 12.7. Finally, in Section 12.8, we discuss an assortment of issues and facilities, including out-of-band data, subnetwork addressing, and the address resolution protocol.

After we have discussed the framework in which the network protocols fit, we shall examine the implementations of several existing network protocols in Chapter 13. A detailed description of the internal data structures and functions of the network layers and protocols can be found in Wright & Stevens [1995].


12.1. Internal Structure

The network subsystem is logically divided into three layers. These three layers manage the following tasks:

  1. Interprocess data transport

  2. Internetwork addressing and message routing

  3. Transmission-media support

The first two layers are made up of modules that implement communication protocols. The software in the third layer generally includes a protocol sublayer, as well as one or more network device drivers.

The topmost layer in the network subsystem is termed the transport layer. The transport layer must provide an addressing structure that permits communication between sockets and any protocol mechanisms necessary for socket semantics, such as reliable data delivery. The second layer, the network layer, is responsible for the delivery of data destined for remote transport or for network-layer protocols. In providing internetwork delivery, the network layer must manage a private routing database or use the systemwide facility for routing messages to their destination host. The bottom layer, the network-interface layer, or link layer, is responsible for transporting messages between hosts connected to a common transmission medium. The network-interface layer is mainly concerned with driving the transmission media involved and doing any necessary link-level protocol encapsulation and decapsulation.

The transport, network, and network-interface layers of the network subsystem correspond to the transport, network, and link layers of the ISO Open Systems Interconnection Reference Model [ISO, 1984], respectively. The internal structure of the networking software is not directly visible to users. Instead, all networking facilities are accessed through the socket layer described in Chapter 11. Each communication protocol that permits access to its facilities exports a set of user request routines to the socket layer. These routines are used by the socket layer in providing access to network services.

The layering described here is a logical layering. The software that implements network services may use more or fewer communication protocols according to the design of the network architecture being supported. For example, raw sockets often use a null implementation at one or more layers. At the opposite extreme, tunneling of one protocol through another uses one network protocol to encapsulate and deliver packets for another protocol and involves multiple instances of some layers.

Data Flow

Early versions of BSD were used as end systems in a network. They were either the source or destination of communication. Although many installations used a workstation as a departmental gateway, dedicated hardware was used to perform the more complex tasks of bridging and routing. At the time of the original design and implementation of the networking subsystem, the possibility of securing data by encrypting packets was still far in the future. Since that initial design, many different uses have been made of the code. Due to the increase in general processor performance, it is now fairly simple to build a bridge or router out of stock parts and the advent of specialized cyptographic coprocessors has made packet encryption practical in a home, cafe, or office environment. These facts conspire to make a discussion of data flow within the network subsystem more complex than it once was.

While the ideas are complex, the code is even more so. Several different implementors have added their particular flavor of change to make their particular application perform in the way they wished, and these changes have led to an overly complex implementation. The basic idea to keep in mind is that there are only four real paths through a network node:

Inbound | Destined for a user-level application.
Outbound | From user-level application to the network.
Forward | Whether bridged or routed the packets are not for this node but to be sent on to another network or host.
Error | A packet has arrived that requires the network subsystem itself to send a response without the involvement of a user level application.

Inbound data received at a network interface flow upward through communication protocols until they are placed in the receive queue of the destination socket. Outbound data flow down to the network subsystem from the socket layer through calls to the transport-layer modules that support the socket abstraction. The downward flow of data typically is started by system calls. Data flowing upward are received asynchronously and are passed from the network-interface layer to the appropriate communication protocol through per-protocol input message queues, as shown in Figure 12.1 (on page 476). The system handles inbound network traffic by splitting the processing of packets between the network driver's upper and lower halves. The lower half of the driver runs in the device's interrupt context and handles the physical interrupts from the device and manages the device's memory. The upper half of the driver runs as an interrupt thread and can either queue packets for the network thread swi_net or process them to completion. FreeBSD 5.2 queues all packets by default. The fine-grained locking of the network code done to support symmetric multiprocessing in FreeBSD has made it possible to preempt interrupt threads that are processing packets without negative side effects, which means that interactive performance does not suffer when a system is under a high network load. A third alternative for packet processing is to have the system poll for packets, and this polling has been added as an experimental feature, but it currently is used only with a few devices and will not be discussed here.

Figure 12.1. Example of upward flow of a data packet in the network subsystem. Key: ETHER—Ethemet header; IP—Internet Protocol header; TCP—Transmission Control Protocol header.

Once the packet has been queued by the device's interrupt thread, the swi_net thread is responsible for handling the packet. This handler is a thread in the kernel whose sole job is to read packets off the queue, and if a message received by a communication protocol is destined for a higher-level protocol, this protocol is invoked directly. If the message is destined for another host (i.e., following the forward path) and the system is configured as a router, the message may be returned to the network-interface layer for retransmission.

Communication Protocols

A network protocol is defined by a set of conventions, including packet formats, states, and state transitions. A communication-protocol module implements a protocol and is made up of a collection of procedures and private data structures.

Protocol modules are described by a protocol-switch structure that contains the set of externally visible entry points and certain attributes, shown in Figure 12.2. The socket layer interacts with a communication protocol solely through the latter's protocol-switch structure, recording the address of the structure in the socket's so_proto field. This isolation of the socket layer from the networking subsystem is important in ensuring that the socket layer provides users with a consistent interface to all the protocols supported by a system. When a socket is created, the socket layer looks up the domain for the protocol family to find the array of protocol-switch structures for the family (see Section 11.4). A protocol is selected from the array based on the type of socket supported (the pr_type field) and optionally a specific protocol number (the pr_protocol field). The protocol switch has a back pointer to the domain (pr_domain). Within a protocol family, every protocol capable of supporting a socket directly (for example, most transport protocols) must provide a protocol-switch structure describing the protocol. Lower-level protocols such as network-layer protocols may also have protocol-switch entries, although whether they do can depend on conventions within the protocol family.

Figure 12.2. Protocol-switch structure.

Before a protocol is first used, the protocol's initialization routine is invoked. Thereafter, the protocol will be invoked for timer-based actions every 200 milliseconds if the pr_fasttimo() entry is present, and every 500 milliseconds if the pr_slowtimo() entry point is present. In general, protocols use the slower timer for most timer processing; the major use of the fast timeout is for delayed-acknowledgment processing in reliable transport protocols. The pr_drain() entry is provided so that the system can notify the protocol if the system is low on memory and would like any noncritical data to be discarded. Finally, the pr_pfil() entry gives a hook for packet-filtering calls, which allow a system administrator to drop or modify packets as they are processed by the networking subsystem.

Protocols may pass data between their layers in chains of mbufs (see Section 11.3) using the pr_input() and pr_output() routines. The pr_input() routine passes data up toward the user, whereas the pr_output() routine passes data down toward the network. Similarly, control information passes up and down via the pr_ctlinput() and pr_ctloutput() routines. The table of user request routines, pr_usrreqs(), is the interface between a protocol and the socket level; it is described in detail in Section 12.2.

In general, a protocol is responsible for storage space occupied by any of the arguments passed downward via these procedures and must either pass the space onward or dispose of it. On output, the lowest level reached must free space passed as arguments; on input, the highest level is responsible for freeing space passed up to it. Auxiliary storage needed by protocols is allocated from the mbuf pool. This space is used temporarily to formulate messages or to hold variable-sized socket addresses. (Some protocols also use mbufs for data structures such as state control blocks, although many such uses have been converted to use malloc() directly.) Mbufs allocated by a protocol for private use must be freed by that protocol when they are no longer in use.

The pr_flags field in a protocol's protocol-switch structure describes the protocol's capabilities and certain aspects of its operation that are pertinent to the operation of the socket level; the flags are listed in Table 12.1. Protocols that are connection based specify the PR_CONNREQUIRED flag, so that socket routines will never attempt to send data before a connection has been established. If the PR_WANTRCVD flag is set, the socket routines will notify the protocol when the user has removed data from a socket's receive queue. This notification allows a protocol to implement acknowledgment on user receipt and also to update flow-control information based on the amount of space available in the receive queue. The PR_ADDR field indicates that any data placed in a socket's receive queue by the protocol will be preceded by the address of the sender. The PR_ATOMIC flag specifies that each user request to send data must be done in a single protocol send request; it is the protocol's responsibility to maintain record boundaries on data to be sent. This flag also implies that messages must be received and delivered to processes atomically. The PR_RIGHTS flag indicates that the protocol supports the transfer of access rights; this flag is currently used by only those protocols in the local communication domain. Connection-oriented protocols that allow the user to set up, send data, and tear down a connection all in a single sendto call have the PR_IMPLOPCL flag set. The PR_LASTIIDR flag is used by secure protocols, such as IPSec, where several headers must be processed to get at the actual data.

Table 12.1. Protocol flags.

Flag

Description

PR_ATOMIC

messages sent separately, each in a single packet

PR_ADDR

protocol presents address with each message

PR_CONNREQUIRED

connection required for data transfer

PR_WANTRCVD

protocol notified on user receipt of data

PR_RIGHTS

protocol supports passing access rights

PR_IMPLOPCL

implied open/close

PR_LASTHDR

used by secure protocols to check for last header

Network Interfaces

Each network interface configured in a system defines a link-layer path through which messages can be sent and received. A link-layer path is a path that allows a message to be sent via a single transmission to its destination, without network-level forwarding. Normally, a hardware device is associated with this interface, although there is no requirement that one be (e.g., all systems have a software loopback interface). In addition to manipulating the hardware device, a network-interface module is responsible for encapsulation and decapsulation of any link-layer protocol header required to deliver a message to its destination. For common interface types, the link-layer protocol is implemented in a separate sublayer that is shared by various hardware drivers. The selection of the interface to use in sending a packet is a routing decision carried out at the network-protocol layer. An interface may have addresses in one or more address families. Each address is set at boot time using an ioctl system call on a socket in the appropriate domain; this operation is implemented by the protocol family after the network interface verifies the operation with an ioctl entry point provided by the network interface. The network-interface abstraction provides protocols with a consistent interface to all hardware devices that may be present on a machine.

An interface and its addresses are defined by the structures shown in Figure 12.3 (on page 480). As interfaces are found at startup time, the ifnet structures are initialized and are placed on a linked list. The network-interface module generally maintains the ifnet interface data structure as part of a larger structure that also contains information used in driving the underlying hardware device. Similarly, the ifaddr interface address structure is often part of a larger structure containing additional protocol information about the interface or its address. Because network socket addresses are variable in size, each protocol is responsible for allocating the space referenced by the address, mask, and broadcast or destination address pointers in the ifaddr structure.

Figure 12.3. Network-interface data structures.

[View full size image]

Each network interface is identified in two ways: a character string identifying the driver plus a unit number for the driver (e.g. fxpO) and a binary systemwide index number. The index is used as a shorthand identifier—for example, when a route that refers to the interface is established. As each interface is found during system startup, the system creates an array of pointers to the ifnet structures for the interfaces. It can thus locate an interface quickly given an index number, whereas the lookup using a string name is less efficient. Some operations, such as interface address assignment, name the interface with a string for the user's convenience because performance is not critical. Other operations, such as route establishment, pass a newer style of identifier that can use either a string or an index. The new identifier uses a sockaddr structure in a new address family, AF_LINK, indicating a link-layer address. The family-specific version of the structure is a sockaddr_dl structure, shown in Figure 12.4, which may contain up to three identifiers. It includes an interface name as a string plus a length, with a length of zero denoting the absence of a name. It also includes an interface index as an integer, with a value of zero indicating that the index is not set. Finally, it may include a binary link-level address, such as an Ethernet address, and the length of the address. An address of this form is created for each network interface as the interface is configured by the system and is returned in the list of local addresses for the system along with network protocol addresses (see later in this subsection). Figure 12.4 shows a structure describing an Ethernet interface that is the first interface on the system; the structure contains the interface name, the index, and the link-layer (Ethernet) address.

Figure 12.4. Link-layer address structure. The box on the left names the elements of the sockaddr_dl structure. The box on the right shows sample values for these elements for an Ethernet interface. The sdl_data array may contain a name (if sdl_nlen is nonzero, a link-layer address (if sdl_alen is nonzero), and an address selector (if sdl_slen is nonzero). For an Ethernet, sdl_data contains a three-character name followed by a unit number, fxpO, followed by a 6-byte Ethernet address.

The interface data structure includes an if_data structure, which contains the externally visible description of the interface. It includes the link-layer type of the interface, the maximum network-protocol packet size that is supported, and the sizes of the link-layer header and address. It also contains numerous statistics, such as packets and bytes sent and received, input and output errors, and other data required by network-management protocols.

The state of an interface and certain externally visible characteristics are stored in the if_flags field described in Table 12.2 (on page 482). The first set of flags characterizes an interface. If an interface is connected to a network that supports transmission of broadcast messages, the IFF_BROADCAST flag will be set, and the interface's address list will contain a broadcast address to be used in sending and receiving such messages. If an interface is associated with a point-to-point hardware link (e.g., a leased line circuit), the IFF_POINTOPOINT flag will be set, and the interface's address list will contain the address of the host on the other side of the connection. (Note that the broadcast and point-to-point attributes are mutually exclusive.) These addresses and the local address of an interface are used by network-layer protocols in filtering incoming packets. The IFF_MULTICAST flag is set by interfaces that support multicast packets in addition to IFF_BROADCAST. Multicast packets are sent to one of several group addresses and are intended for all members of the group.

Table 12.2. Network interface flags.

Flag

Description

IFF_UP

interface is available for use

IFF_BROADCAST

broadcast is supported

IFF_DEBUG

enable debugging in the interface software

IFF_LOOPBACK

this is a software loopback interface

IFF_POINTOPOINT

interface is for a point-to-point link

IFF_SMART

interface manages its own routes

IFF_RUNNING

interface resources have been allocated

IFF_NOARP

interface does not support ARP

IFF_PROMISC

interface receives packets for all destinations

IFF_ALLMULTI

interface receives all multicast packets

IFF_OACTIVE

interface is busy doing output

IFF_SIMPLEX

interface cannot receive its own transmissions

IFF_LINKO

link-layer specific

IFF_LINK1

link-layer specific

IFF_LINK2

link-layer specific

IFF_MULTICAST

multicast is supported

IFF_POLLING

interface is in polling mode

IFF_PPROMISC

user-requested promiscuous mode

IFF_MONITOR

user-requested monitor mode

IFF_STATICARP

interface only uses static ARP

Additional interface flags describe the operational state of an interface. An interface sets the IFF_RUNNING flag after it has allocated system resources and has posted an initial read on the device that it manages. This state bit avoids multiple allocation requests when an interface's address is changed. The IFF_UP flag is set when the interface is configured and is ready to transmit messages. The IFF_OACTIVE flag is used to coordinate between the if_output and if_start routines, described later in this subsection; it is set when no additional output may be attempted. The IFF_PROMISC flag is set by network-monitoring programs to enable promiscuous reception: when they wish to receive packets for all destinations rather than for just the local system. Packets addressed to other systems are passed to the monitoring packet filter but are not delivered to network protocols. The IFF_ALLMULTI flag is similar, but it applies to only multicast packets and is used by multicast forwarding agents. The IFF_SIMPLEX flag is set by Ethernet drivers whose hardware cannot receive packets that they send. Here, the output function simulates reception of broadcast and (depending on the protocol) multicast packets that have been sent. Finally, the IFF_DEBUG flag can be set to enable any optional driver-level diagnostic tests or messages. Three flags are defined for use by individual link-layer drivers (IFF_LINKO, IFF_LINK1, and IFF_LINK2). They can be used to select link-layer options, such as Ethernet medium type.

Interface addresses and flags are set with ioctl requests. The requests specific to a network interface pass the name of the interface as a string in the input data structure, with the string containing the name for the interface type plus the unit number. Either the SIOCSIFADDR request or the SIOCAIFADDR request is used initially to define each interface's addresses. The former sets a single address for the protocol on this interface. The latter adds an address, with an associated address mask and broadcast address. It allows an interface to support multiple addresses for the same protocol. In either case, the protocol allocates an ifaddr structure and sufficient space for the addresses and any private data and links the structure onto the list of addresses for the network interface. In addition, most protocols keep a list of the addresses for the protocol. The result appears somewhat like a two-dimensional linked list, as shown in Figure 12.5. An address can be deleted with the SIOCDIFADDR request.

Figure 12.5. Network-interface and protocol data structures. The linked list of ifnet structures appears on the left side of the figure. The ifaddr structures storing the addresses for each interface are on a linked list headed in the ifnet structure and shown as a horizontal list. The ifaddr structures for most protocols are linked together as well, shown in the vertical lists headed by pf1_addr and pf2_addr.

The SIOCSIFFLAGS request can be used to change an interface's state and to do site-specific configuration. The destination address of a point-to-point link is set with the SIOCSIFDSTADDR request. Corresponding operations exist to read each value. Protocol families also can support operations to set and read the broadcast address. Finally, the SIOCGIFCONF request can be used to retrieve a list of interface names and protocol addresses for all interfaces and protocols configured in a running system. Similar information is returned by a newer mechanism based on the sysctl system call with a request in the routing protocol family (see Sections 12.5 and 14.6). These requests permit developers to construct network processes, such as a routing daemon, without detailed knowledge of the system's internal data structures.

Each interface has a queue of packets to be transmitted and routines used for initialization and output. The if_output() routine accepts a packet for transmission and normally handles link-layer encapsulation and queueing that are independent of the specific hardware driver in use. If the IFF_OACTIVE flag is not set, the output routine may then invoke the driver's if_start() function to begin transmission. The start function then sets the IFF_OACTIVE flag if it is unable to accept additional packets for transmission; the flag will be cleared when transmission completes. The if_done() entry point is provided as a callback function for use when the output queue is emptied. This facility is not yet used, but it is intended to support striping of data for a single logical interface across multiple physical interfaces.

An interface may also specify a watchdog timer routine and a timer value that (if it is nonzero) the system will decrement once per second, invoking the timer routine when the value expires. The timeout mechanism is typically used by interfaces to implement watchdog schemes for unreliable hardware and to collect statistics that reside on the hardware device.


12.2. Socket-to-Protocol Interface

The interface from the socket routines to the communication protocols is through the user request table pr_usrreqs() and the pr_ctloutput() routine, which are defined in the protocol-switch table for each protocol. When the socket layer requires services of a supporting protocol, it makes a call to one of the functions in Table 12.3. The control-output routine implements the getsockopt and setsockopt system calls; the user-request routines are used for all other operations. Calls to pr_ctloutput() specify SOPT_GET to get the current value of an option, or SOPT_SET to set the value of an option.

Table 12.3. pr_usrreqs() routine table.

Entry point

Description

pru_abort()

abort connection and detach

pru_accept()

accept connection from peer

pru_attach()

attach protocol to socket

pru_bind()

bind name to socket

pru_connect()

establish connection to peer

pru_connect2()

connect two sockets

pru_control()

control protocol operation (ioctl)

pru_detach()

detach protocol from socket

pru_disconnect()

disconnect from peer

pru_listen()

listen for connections

pru_peeraddr()

fetch peer's address

pru_rcvd()

have taken data; more room now

pru_rcvoob()

retrieve out-of-band data

pru_send()

send these data

pru_sense()

sense socket status (fstat)

pru_shutdown()

will not send any more data

pru_sockaddr()

fetch socket's address

pru_sosend()

fast path for sosend

pru_soreceive()

fast path for sereceive

pru_sopoll()

fast path for sopoll

Protocol User-Request Routines

Calls to the user-request routines have a routine-specific signature, but the first argument is always a pointer to a socket structure, which specifies the socket for which the operation is intended. An mbuf data chain is supplied for output operations and for certain other operations where a result is to be returned. A pointer to a sockaddr structure is supplied for address-oriented requests, such as pru_bind(), pru_connect(), and pru_send() (when an address is specified—e.g., the sendto call). Where it is used, the control parameter is a pointer to an optional mbuf chain containing protocol-specific control information passed via the sendmsg call. Each protocol is responsible for disposal of the data mbuf chains on output operations. A nonzero return value from a user-request routine indicates an error number that should be passed to higher-level software. A description of each of the possible requests follows:

  • pru_attach(): attach protocol to socket. When a protocol is first bound to a socket (with the socket system call), the protocol module's pru_attach() routine is called. It is the responsibility of the protocol module to allocate any resources necessary. The attach routine will always precede any of the other operations and will occur only once per socket.

  • pru_detach(): detach protocol from socket. This operation is the inverse of the attach routine and is used at the time that a socket is deleted. The protocol module may deallocate any resources that it allocated for the socket in a previous pru_attach() call.

  • pru_bind(): bind address to socket. When a socket is initially created, it has no address bound to it. This routine binds an address to an existing socket. The protocol module must verify that the requested address is valid and is available for use.

  • pru_listen(): listen for incoming connections. The listen request routine indicates that the user wishes to listen for incoming connection requests on the associated socket. The protocol module should make any state changes needed to meet this request (if possible). A call to the listen routine always precedes any request to accept a connection.

  • pru_connect(): connect socket to peer. The connect request routine indicates that the user wants to establish an association. The addr parameter describes the peer to which a connection is desired. The effect of a connect request may vary depending on the protocol. Stream protocols use this request to initiate establishment of a network connection. Datagram protocols simply record the peer's address in a private data structure, where they use it as the destination address of all outgoing packets and as a source filter for incoming packets. There are no restrictions on how many times a connect routine may be used after an attach, although most stream protocols allow only one connect call.

  • pru_accept(): accept pending connection. Following a successful listen request and the arrival of one or more connections, this routine is called to indicate that the user is about to accept a socket from the queue of sockets ready to be returned. The socket supplied as a parameter is the socket that is being accepted; the protocol module is expected to fill in the supplied buffer with the address of the peer connected to the socket.

  • pru_disconnect(): disconnect connected socket. This routine eliminates an association created with the connect routine. It is used with datagram sockets before a new association is created; it is used with stream protocols only when the socket is closed.

  • pru_shutdown(): shut down socket data transmission. This call indicates that no more data will be sent. The protocol may, at its discretion, deallocate any data structures related to the shutdown or the protocol may leave all of that work for its pru_detach() routine. The module may also notify a connected peer of the shutdown at this time.

  • pru_rcvd(): data were received by user. This routine is called only if the protocol entry in the protocol-switch table includes the PR_WANTRCVD flag. When the socket layer removes data from the receive queue and passes them to the user, this routine will be called in the protocol module. This routine may be used by the protocol to trigger acknowledgments, refresh windowing information, initiate data transfer, and so on. This routine is also called when an application attempts to receive data on a socket that is in the confirming state, indicating that the protocol must accept the connection request before data can be received (see Section 11.5).

  • pru_send(): send user data. Each user request to send data is translated into one or more calls to the protocol module's pru_send() routine. A protocol may indicate that a single user send request must be translated into a single call to the pru_send() routine by specifying the PR_ATOMIC flag in its protocol description. The data to be sent are presented to the protocol as a chain of mbufs, and an optional address is supplied in the addr parameter. The protocol is responsible for preserving the data in the socket's send queue if it is not able to send them immediately or if it may need them at some later time (e.g., for retransmission). The protocol must eventually pass the data to a lower level or free the mbufs.

  • pru_abort(): abnormally terminate service. This routine effects an abnormal termination of service. The protocol should delete any existing associations.

  • pru_control(): do control operation. The control request routine is called when a user does an ioctl system call on a socket and the ioctl is not intercepted by the socket routines. This routine allows protocol-specific operations to be provided outside the scope of the common socket interface. The cmd parameter contains the actual ioctl request code. The data parameter contains any data relevant to the command being issued and the ifp parameter contains a pointer to a network-interface structure if the ioctl operation pertains to a particular network interface.

  • pru_sense(): sense socket status. The sense request routine is called when the user makes an fstat system call on a socket; it requests the status of the associated socket. This call returns a standard stat structure that typically contains only the optimal transfer size for the connection (based on buffer size, windowing information, and maximum packet size).

  • pru_rcvoob(): receive out-of-band data. This routine requests that any out-of-band data now available are to be returned. An mbuf is passed to the protocol module, and the protocol should either place data in the mbuf or attach new mbufs to the one supplied if there is insufficient space in the single mbuf. An error may be returned if out-of-band data are not (yet) available or have already been consumed. The flags parameter contains any options, such as MSG_PEEK, that should be observed while this request is carried out.

  • pru_sockaddr(): retrieve local socket address. This routine returns the local address of the socket if one has been bound to the socket. The address is returned in the nam parameter, which is a pointer to a sockaddr structure.

  • pru_peeraddr(): retrieve peer socket address. This routine returns the address of the peer to which the socket is connected. The socket must be in a connected state for this request to succeed. The address is returned in the nam parameter, which is a pointer to a sockaddr structure.

  • pru_connect2(): connect two sockets without binding addresses. In this routine, the protocol module is supplied two sockets and is asked to establish a connection between the two without binding any addresses, if possible. The system uses this call in implementing the socketpair system call.

  • pru_fasttimo(): service fast timeout. This routine is called when the fast timeout expires (200ms). The fast-timeout routine cannot be called from the socket layer.

  • pru_slowtimo(): service slow timeout. This routine is called when the slow timeout has expired (500ms). This slow-timeout routine cannot be called from the socket layer.

Protocol Control-Output Routine

A call to the control-output routine is of the form

int (*pr->pr_ctloutput)(
    struct socket *so,
    struct sockopt *sopt);

where so is the socket to be modified and sopt is a socket option structure.

enum sopt_dir { SOPT_GET, SOPT_SET };

struct sockopt {
    enumsopt_dir sopt_dir;
    int sopt_level;
    int sopt_name;
    void   *sopt_val;
    size_t  sopt_valsize;
    struct  thread *sopt_td;
};

The direction is either SOPT_SET to set an option or SOPT_GET to retrieve it. The sopt_level member indicates the layer of software that should interpret the option request. A sopt_level of SOL_SOCKET is specified to control an option at the socket layer. When the option is to be processed by a protocol module below the socket layer, level is set to the appropriate protocol number (the same number used in the socket system call). Each level has its own set of option names; this name is interpreted only by the targeted layer of software. The rest of the structure contains a pointer to the value being passed into or out of the module, the size of the pointed-to data, and a pointer to a thread structure. If the operation takes place wholly inside the kernel, then the pointer to the thread structure is null.

In supporting the getsockopt and setsockopt system calls, the socket layer always invokes the control-output routine of the protocol attached to the socket. To access lower-level protocols, each control-output routine must pass control-output requests that are not for itself downward to the next protocol in the protocol hierarchy. Chapter 13 describes some of the options provided by the protocols in the Internet communication domain.


12.3. Protocol-Protocol Interface

The interface between protocol modules uses the pr_usrreqs() routines as well as the pr_ctloutput() routine. The pr_usrreqs() and pr_ctloutput() routines are used by the socket layer to communicate with protocols.

Although imposing a standard calling convention for all of a protocol's entry points might theoretically permit an arbitrary interconnection of protocol modules, it would be difficult in practice. Crossing of a protocol-family boundary—for example, between IPv4 and IPX—would require a network address to be converted from the format of the caller's domain to the format of the receiver's domain. Consequently, connection of protocols in different communication domains is not generally supported, and calling conventions for the routines listed in the preceding paragraph are typically standardized on a per-domain basis. (However, the system does support encapsulation of packets from one protocol into packets of a protocol in another family to tunnel one protocol through another.)

In this section, we briefly examine the general framework and calling conventions of protocols. In Chapter 13, we examine specific protocols to see how they fit into this framework.

pr_output

Each protocol has a different calling convention for its output routine. This lack of standardization is one of the things that prevents protocol modules from being freely interchanged with each other in arbitrary stacks, such as is done in the STREAMS system [Ritchie, 1984]. Thus far this kind of standardization has not been considered necessary because each protocol stack tends to stand on its own without ever borrowing from others. An arbitrary stacking of protocol modules would also complicate the interpretation of network addresses in each module, since each would have to check to make sure that the address made some sense to them in their domain.

The simplest example of a protocol output routine often uses a calling convention designed to send a single message on a connection; for example,

int (*pr_output)(
    register struct inpcb *inp,
    struct mbuf *msg,
    struct sockaddr *addr
    struct mbuf *control,
    struct thread *td);

would send a message contained in msg on a socket described by protocol control block inp. Special address and control information are passed in addr and control, respectively.

pr_input

Upper-level protocol input routines are usually called by the network software-interrupt task once the network-level protocol has located the protocol identifier. They have stricter conventions than do output routines because they are called via the protocol switch. Depending on the protocol family, they may receive a pointer to a control block identifying the connection, or they may have to locate the control block from information in the received packet. A typical calling convention is

void (*pr_input)(
    struct mbuf *msg,
    int hlen);

In this example, the incoming packet is passed to a transport protocol in an mbuf msg with the network protocol header still in place for the transport protocol to use, as well as the length of the header, hlen, so that the header can be removed. The protocol does the endpoint-level demultiplexing based on information in the network and transport headers.

pr_ctlinput

This routine passes control information (i.e., information that might be passed to the user but does not consist of data) upward from one protocol module to another. The common calling convention for this routine is

void (*pr_ctlinput)(
    int cmd,
    struct sockaddr *addr,
    void* opaque);

The cmd parameter is one of the values shown in Table 12.4. The addr parameter is the remote address to which the condition applies. Many of the requests have been derived from the Internet Control Message Protocol (ICMP) [Postel, 1981] and from error messages defined in the 1822 host (Internet Message Processor) convention [BBN, 1978]. Some protocols may pass additional parameters internally, such as local addresses or more specific information.

Table 12.4. Control-input routine requests.

Request

Description

PRC_IFDOWN

network interface transition to down

PRC_ROUTEDEAD

select new route if possible

PRC_IFUP

network interface has come back up

PRC_QUENCH

some receiver said to slow down

PRC_QUENCH2

DEC congestion bit says slow down

PRC_MSGSIZE

message size forced packet to be dropped

PRC_HOSTDEAD

remote host is down

PRC_HOSTUNREACH

remote host is unreachable

PRC_UNREACH_NET

no route to network

PRC_UNREACH_HOST

no route to host

PRC_UNREACH_PROTOCOL

protocol not supported by destination

PRC_UNREACH_PORT

port number not in use at destination

PRC_UNREACH_SRCFAIL

source routing failed

PRC_REDIRECT_NET

routing redirect for a network

PRC_REDIRECT_HOST

routing redirect for a host

PRC_REDIRECT_TOSNET

routing redirect for type of service and network

PRC_REDIRECT_TOSHOST

routing redirect for type of service and host

PRC_TIMXCEED_INTRANS

packet lifetime expired in transit

PRC_TIMXCEED_REASS

lifetime expired on reassembly queue

PRC_PARAMPROB

header-parameter problem detected

PRC_UNREACH_ADMIN_PROHIB

packet administratively prohibited


12.4. Interface Between Protocol and Network Interface

The lowest layer in the set of protocols that constitutes a protocol family must interact with one or more network interfaces to transmit and receive packets. It is assumed that any routing decisions have been made before a packet is sent to a network interface; a routing decision is necessary to locate any interface at all. Although there are four paths through any network stack, there are only two cases with which we should be concerned in the interaction between protocols and network interfaces: transmission of a packet and receipt of a packet. We shall consider each separately.

Packet Transmission

If a protocol has chosen an interface identified by ifp, a pointer to a network interface structure, the protocol transmits a fully formatted network-level packet with the following call:

int (*if_output)(
    struct ifnet *ifp,
    struct mbuf *msg,
    struct sockaddr *dst,
    struct rtentry *rt);

The output routine for the network interface transmits the packet msg to the protocol address specified in dst or returns an error number. In reality, transmission may not be immediate or successful; typically, the output routine validates the destination address, queues the packet on its send queue, and primes an interrupt-driven routine to transmit the packet if the interface is not busy. For unreliable media, such as Ethernet, successful transmission simply means that the packet has been placed on the cable without a collision. In contrast, a reliable, point-to-point network such as X.25, can guarantee proper delivery of a packet or give an error indication for each packet that was not successfully transmitted. The model employed in the networking system attaches no promise of delivery to the packets presented to a network interface and thus corresponds most closely to the Ethernet. Errors returned by the output routine are only those that can be detected immediately and are normally trivial in nature (network down, no buffer space, address format not handled, etc.). If errors are detected after the call has returned, the protocol is not notified.

When messages are transmitted in a broadcast network such as Ethernet, each network interface must formulate a link-layer address for each outgoing packet. The interface layer must understand each protocol address format that it supports to formulate corresponding link-layer addresses. The network layer for each protocol family selects a destination address for each message and then uses that address to select the appropriate network interface to use. This destination address is passed to the interface's output routine as a sockaddr structure. Presuming that the address format is supported by the interface, the interface must map the destination protocol address into an address for the link-layer protocol associated with the transmission medium that the interface supports. This mapping may be a simple algorithm, it may require a table lookup, or it may require more involved techniques, such as use of the address resolution protocol described in Section 12.8.

Packet Reception

Network interfaces receive packets and dispatch packets to the appropriate network-layer protocol according to information encoded in the link-layer protocol header. Each protocol family must have one or more protocols that constitute the network layer described in Section 12.1. In this system, each network-layer protocol has an input-packet queue assigned to it. Incoming packets received by a network interface are queued in a protocol's input packet queue, and the network thread handles network-layer processing; see Figure 12.6. Similar queues are used to store packets awaiting transmission by network device drivers.

Figure 12.6. Input packets dispatched to protocol input queues.

Several macros are available for manipulating packet queues:

• IF_ENQUEUE ( if q, m) | Place the packet m at the tail of the queue ifq.
• IF_DEQUEUE (ifq, m) | Place a pointer to the packet at the head of queue ifq in m and remove the packet from the queue; m will be zero if the queue is empty.
• IF_PREPEND(ifq, m) | Place the packet m at the head of the queue ifq.

Packet queues have a maximum length associated with them as a simple form of congestion control. The macro IF_QFULL() can be used to determine whether a queue is full; if it is, another macro, IF_DROP(), can then be used to record the event in statistics kept for the queue. Each queue is protected by a mutex so that threads on different processors cannot accidentally interfere with each other's transmissions. Any macro that manipulates the queue first locks the mutex, then makes its change, and finally releases the mutex.

As an example, the following code fragment could be used in a network interface's output routine:

if (IF_QFULL(ifp–>if_snd)) {
    IF_DROP(ifp–>if_snd);
    m_freem(m);      /* discard packet */
    error = ENOBUFS;
} else
    IF_ENQUEUE(ifp–>if_snd, m);

On receiving a packet, a network interface decodes the packet type, strips the link-layer protocol header, records the identity of the receiving interface, and then dispatches the packet to the appropriate link-layer module. Above the link layer each protocol has an input handler routine and input queue, which are registered with the network thread via the netisr_register() routine.

For example, an Ethernet device driver queues its packets in the Ethernet link layer by calling ether_demux(), which then calls netisr_dispatch() to enqueue packets for a particular protocol. netisr_dispatch() is a generic routine that hands packets to protocols. It in turn uses a generic routine if_handoff() to actually enqueue the packet with the following code:

IF_LOCK_(ifq) ;
if (_IF_QFULL(ifq) ) {
    _IF_DROP(ifq);
    IF_UNLOCK(ifq);
    m_freem(m);
    return (0);
}
_IF_ENQUEUE(ifq, m);
IF_UNLOCK(ifq);

Once the packet is placed in the queue, the schednetisr() macro schedules the network thread so that it will carry on the processing of the packet.

Entries on a protocol's input queue are mbuf chains with a valid packet header containing the packet's length and a pointer to the network interface on which the packet was received. The pointer to the interface has many potential uses, such as deciding when to generate routing redirect messages. The protocol's input routine does not dequeue the packet but is handed an mbuf pointing to the packet by the network thread. It is the network thread that dequeues packets and then calls the protocol's input routine.

While an entry is being dequeued from an input queue, the network thread blocks all other packets from entering the protocol by locking the network thread mutex. The lock is held to ensure that pointers in the queue data structure are not altered. Once a message is dequeued, it is processed; if there is information in the packet for a higher-level protocol, the message is passed upward.


12.5. Routing

The networking system was designed for a heterogenous network environment, in which a collection of local-area networks are connected at one or more points through routers, as shown in the example in Figure 12.7. Routers are nodes with multiple network interfaces, one on each local-area or long-haul network. In such an environment, issues related to packet routing are important. Certain of these issues, such as congestion control, are handled simplistically in FreeBSD (see Section 12.6). For others, the network system provides simple mechanisms on which more involved policies can be implemented. These mechanisms ensure that, as these problems become better understood, their solutions can be incorporated into the system. Note that at the time of the original design of this part of the system, a network node that forwarded network-level packets was generally known as a gateway. The current term is router. We use both terms interchangeably, in part because the system data structures continue to use the name gateway.

Figure 12.7. Example of the topology for which routing facilities were designed.

This section describes the facilities provided for packet routing. The routing facilities were designed for use by singly connected and multiply connected hosts, as well as for routers. There are several components involved in routing, illustrated in Figure 12.8. The design of the routing system places some components within the kernel and others at user level. Routing is actually an overbroad term. In a complex, modern network there are two major components to a routing system. The gathering and maintenance of route information (i.e., which interfaces are up, what the costs are in terms of using particular links, etc.) as well as the implementation of routing policies (which interfaces can be used to forward traffic in an administrative sense) are handled at user level by routing daemons. The actual forwarding of packets, which is the selection of the interface on which a packet will be sent, is handled by the kernel.

Figure 12.8. Routing design.

The forwarding mechanism is a simple lookup that provides a first-hop route (a specific network interface and immediate destination) for each outbound packet. The current design places enough information in the kernel for packets to be sent on their way without external help; all other components are outside the kernel. User-level routing daemons communicate with the kernel via a routing socket to manipulate the kernel forwarding table and to listen for internal changes such as interfaces being brought up or down. Each of these components is described in this section.

Kernel Routing Tables

The kernel routing mechanism implements a routing table for looking up first-hop routes (or next hop, when forwarding packets). It includes two distinct portions: a data structure describing each specific route (a routing entry) and a lookup algorithm to find the correct route for each possible destination. This subsection describes the entries in the routing table, and the next subsection explains the lookup algorithm. A destination is described by a sockaddr structure with an address family, a length, and a value. Routes are classified in the following ways: as either host or network routes and as either direct or indirect. The host-network distinction determines whether the route applies to a specific host or to a group of hosts with a portion of their addresses in common—usually a prefix of the address. For host routes, the destination address of a route must exactly match the desired destination; the address family, length, and bit pattern of the destination must match those in the route. For network routes, the destination address in the route is paired with a mask. The route matches any address that contains the same bits as the destination in the positions indicated by bits set in the mask. A host route is a special case of a network route, in which all the mask bits are set, and thus no bits are ignored in the comparison. Another special case is a wildcard route—a network route with an empty mask. Such a route matches every destination and serves as a default route for destinations not otherwise known. This fallback network route is usually pointed to a router that can then make more informed routing decisions.

The other major distinction between types of routes is either direct or indirect. A direct route is one that leads directly to the destination: The first hop of the path is the entire path, and the destination is on a network shared with the source. Most routes are indirect: The route specifies a router on a local network that is the first-hop destination for the packet. Much of the literature (especially for Internet protocols) refers to a local-remote decision, where an implementation checks first whether a destination is local to an attached network or is remote. In the first case, a packet is sent locally (via the link layer) to the destination; in the latter case, it is sent to a router that can forward it to the destination. In the FreeBSD implementation, the local-remote decision is made as part of the routing lookup. If the best route is direct, then the destination is local. Otherwise, the route is indirect, the destination is remote, and the route entry specifies the router for the destination. In either case, the route specifies only the first-hop gateway—a link-level interface to be used in sending packets—and the destination for the packets in this hop if different from the final destination. This information allows a packet to be sent via a local interface to a destination directly reachable via that interface—either the final destination or a router on the path to the destination. This distinction is needed when the link-layer encapsulation is done. If a packet is destined for a peer that is not directly connected to the source, the packet header will contain the address of the eventual destination, whereas the link-layer protocol header will address the intervening router.

The network system maintains a set of routing tables that is used by protocols in selecting a network interface to use in delivering a packet to its destination. These tables are composed of entries of the form shown in Table 12.5.

Table 12.5. Elements of a routing-table entry (rtentry) structure.

Element

Description

rt_nodes[2]

internal and leaf radix nodes (with references to destination and mask)

rt_gateway

address of the router/next hop

rt_refent

reference count

rt_flags

flags; see Table 12.6

rt_ifp

reference to interface, ifnet

rt_ifa

reference to interface address, ifaddr

rt_genmask

mask for cloning

rt_llinfo

pointer to link-layer private data

rt_rmx

route metrics (e.g., MTU)

rt_gwroute

if indirect, route to gateway

rt_parent

cloning parent of this route

rt_output

output routine for this route/interface, unused

rt_mtx

mutex for locking the entry (kernel only)

Table 12.6. Route entry flags.

Flag

Description

RTF_UP

route is valid

RTF_GATEWAY

destination is a gateway

RTF_HOST

host entry (net otherwise)

RTF_REJECT

host or net unreachable

RTF_DYNAMIC

created dynamically (by redirect)

RTF_MODIFIED

modified dynamically (by redirect)

RTF_DONE

message confirmed

RTF_MASK

subnet mask present

RTF_CLONING

generate new routes on use

RTF_XRESOLVE

external daemon resolves name

RTF_LLINFO

generated by link layer

RTF_STATIC

manually added by administrator

RTF_BLACKHOLE

just discard packets (during updates)

RTF_PROTO1

protocol-specific routing flag

RTF_PROTO2

protocol-specific routing flag

RTF_PROTO3

protocol-specific routing flag

RTF_WASCLONED

route generated through cloning

RTF_LOCAL

route represents a local address

RTF_BROADCAST

route represents a beast address

RTF_MULTICAST

route represents an mcast address

Routing entries are stored in an rtentry structure, which contains a reference to the destination address and mask (unless the route is to a host, in which case the mask is implicit). The destination address, the address mask, and the gateway address are variable in size and thus are placed in separately allocated memory. Routing entries also contain a reference to a network interface, a set of flags that characterize the route, and optionally a gateway address. The flags indicate a route's type (host or network, direct or indirect) and the other attributes shown in Table 12.6 (on page 498). The route entry also contains a field for use by the link-layer driver, a set of metrics, and a mutex for locking the entry. The RTF_HOST flag in a routing-table entry indicates that the route applies to a single host, using an implicit mask containing all the bits of the address. The RTF_GATEWAY flag in a routing-table entry indicates that the route is to a router and that the link-layer header should be filled in from the rt_gateway field, instead of from the final destination address. The route entry contains a field that can be used by the link layer to cache a reference to the direct route for the router. The RTF_UP flag is set when a route is installed. When a route is removed, the RTF_UP flag is cleared, but the route entry is not freed until all users of the route have noticed the failure and have released their references. The route entry contains a reference count because it is allocated dynamically and cannot be freed until all references have been released. The RTF_CLONING flag indicates that a route is a generic route that must be cloned and made more specific before use. This flag is usually used for link-layer routes that apply to a directly attached network, and the cloned routes are generally host routes for hosts on that network that contain some link-level information about that host. When a route is cloned, an external agent may be invoked to complete the link-layer information needed for a destination. Other flags (RTF_REJECT and RTF_BLACKHOLE) mark the destination of the route as being unreachable, causing either an error or a silent failure when an attempt is made to send to the destination. Reject routes are useful when a router receives packets for a cluster of addresses from the outside but may not have routes for all hosts or networks in the cluster at all times. It is undesirable for packets with unreachable destinations to be sent outside the cluster via a default route because the default router would send back such packets for delivery within the cluster. Black-hole routes are used during routing transients when a new route may become available shortly.

Many connection-oriented protocols wish to retain information about the characteristics of a particular network path. Some of this information can be estimated dynamically for each connection, such as the round-trip time or path MTU. It is useful to cache such information so that the estimation does not need to begin anew for each connection [Mogul & Deering, 1990]. The routing entry contains a set of route metrics stored in a rt_metrics_lite structure that may be set externally or may be determined dynamically by the protocols. These metrics include the maximum packet size for the path, called the maximum transmission unit (MTU); the lifetime for the route; and the number of packets that have been sent using this route. In versions of FreeBSD prior to 5.2 path MTU discovery was implemented using a much larger structure, called rt_metrics. Most of the fields contained in the rt_metrics structure were related only to TCP, and so they were moved from the routing entry structure into the TCP code.

When a route is added or created by cloning, and when a route is deleted, the link layer is called via the ifa_rtrequest entry point stored in the ifaddr structure for this interface address. The link layer can allocate private storage associated with the route entry. This feature is used with direct routes to networks that are marked as cloning routes; the link layer can use this mechanism to manage link-layer address-translation information for each host. The address translation can be arranged within the system—it can be handled outside the kernel when the RTF_XRESOLVE flag is set.

Routing Lookup

Given a set of routing entries describing various destinations, from specific hosts to a wildcard route, a routing lookup algorithm is required. The lookup algorithm in FreeBSD uses a modification of the radix search trie [Sedgewick, 1990]. (The initial design was to use a PATRICIA search, also described in Sedgewick [1990], which differs only in the details of storage management.) The radix search algorithm provides a way to find a bit string, such as a network address, in a set of known strings. Although the modified search was implemented for routing lookups, the radix code is implemented in a more general way so that it can be used for other purposes. For example, the filesystem code uses a radix tree to manage information about clients to which filesystems can be exported. Each kernel route entry begins with the data structures for the radix tree, including an internal radix node and a leaf node that refers to the destination address and mask.

The radix search algorithm uses a binary tree of nodes beginning with a root node for each address family. Figure 12.9 (on page 500) shows an example radix tree. A search begins at the root node and descends through some number of internal nodes until a leaf node is found. Each internal node requires a test of a specific bit in the string, and the search descends in one of two directions depending on the value of that bit. The internal nodes contain an index of the bit to be tested, as well as a precomputed byte index and mask for use in the test. A leaf node is marked with a bit index of -1, which terminates the search. For example, a search for the address 127.0.0.1 (the loopback address) with the tree in Figure 12.9 would start at the head and would branch left when testing bit 0, right at the node for bit 1, and right on testing bit 31. This search leads to the leaf node containing a host route specific to that host; such a route does not contain a mask but uses an implicit mask with all bits set.

Figure 12.9. Example radix tree. This simplified example of a radix tree contains routes for the IPv4 protocol family, which uses 32-bit addresses. The circles represent internal nodes, beginning with the head of the tree at the top. The bit position to be tested is shown within the circle. Leaf nodes are shown as rectangles containing a key (a destination address, listed as four decimal bytes separated by dots) and the corresponding mask (in hexadecimal). Some interior nodes are associated with masks found lower in the tree, as indicated by dashed arrows.

[View full size image]

This lookup technique tests the minimum number of bits required to distinguish among a set of bit strings. Once a leaf node is found, either it specifies the specific bit string in question or that bit string is not present in the tree. This algorithm allows a minimal number of bits to be tested in a string to lookup an unknown, such as a host route; however, it does not provide for partial matching as required by a routing lookup for a network route. Thus, the routing lookup uses a modified radix search, in which each network route includes a mask, and nodes are inserted into the tree such that longer masks are found earlier in the search [Sklower, 1991]. Interior nodes for subtrees with a common prefix are marked with a mask for that prefix. (Masks generally select a prefix from an address, although the mask does not need to specify a contiguous portion of the address.) As the routing lookup proceeds, the internal nodes that are passed are associated with masks that increase in specificity. If the route that is found at the leaf after the lookup is a network route, the destination is masked before comparison with the key, thus matching any destination on that network. If the leaf node does not match the destination, one of the interior nodes visited during the route lookup should refer to the best match. After a lookup does not find a match at the leaf node, the lookup procedure iterates backward through the tree, using a parent pointer in each node. At each interior node that contains a mask, a search is made for the part of the destination under that mask from that point. For example, a search for the address 128.32.33.7 in the tree in Figure 12.9 would test bits 0, 18, and 29 before arriving at the host route on the right (128.32.33.5). Because this address is not a match, the search moves up one level, where a mask is found. The mask is a 24-bit prefix, and it is associated with the route to 128.32.33.0, which is the best match. If the mask was not a prefix (in the code, a route with a mask specifying a prefix is called a normal route), a search would have been required for the value 128.32.33.7 starting from this point.

The first match found is the best match for the destination; that is, it has the longest mask for any matching route. Matches are thus found by a combination of a radix search, testing 1 bit per node on the way down the tree, plus a full comparison under a mask at the leaf node. If the leaf node (either host or network) does not match, the search backtracks up the tree, checking each parent with a mask until a match is found. This algorithm avoids a complete comparison at each step when searching down the tree, which would eliminate the efficiency of the radix search algorithm. It is optimized for matches to routes with longer masks and performs least efficiently when the best match is the default route (the route with the shortest mask).

Another complication of using a radix search is that a radix tree does not allow duplicated keys. There are two possible reasons for a key to be duplicated in the tree: Either multiple routes exist to the same destination or the same key is present with different masks. The latter case is not a complete duplicate, but the two routes would occupy the same location in the tree. The routing code does not support completely duplicate routes, but it supports multiple routes that differ in only the mask. When the addition of a route causes a key to be duplicated, the affected routes are chained together from a single leaf node. The routes are chained in order of mask significance, most specific mask first. If the masks are contiguous, longer masks are considered to be more specific (with a host route considered to have the longest possible mask). If a routing lookup visits a node with a duplicated key when doing a masked comparison (either at the leaf node or while moving back up the tree), the comparison is repeated for each duplicate node on the chain, with the first successful comparison producing the best match.

As we noted, FreeBSD does not support multiple routes to the same destination (identical key and mask). The main reason to support multiple paths would be to allow the load to be split among the paths but interleaving of traffic across the available paths would often be suboptimal. A better design would be to add a pointer to an output function in each route. Most routes would copy the output pointer for the interface used by the route. Routes for which multiple paths were available would be represented by a virtual route containing references to the individual routes, which would not be placed in the radix tree. The virtual route would interpose an intermediate output function that would distribute packets to the output functions for the individual routes. This scheme would allow good packet interleaving even when a path was used by a single connection. Although the output routine field has been added to the rtentry structure, it is not in use in the code as of FreeBSD 5.2.

Routing Redirects

A routing redirect is a control request from a protocol to the routing system to modify an existing routing-table entry or to create a new routing-table entry. Protocols usually generate such requests in response to redirect messages that they receive from routers. Routers generate redirect messages when they recognize that a better route exists for a packet that they have been asked to forward. For example, if two hosts, A and B, are on the same network, and host A sends a packet to host B via a router C, then C will send a redirect message to A indicating that A should send packets to B directly.

On hosts where exhaustive routing information is too expensive to maintain (e.g., SOHO routers, DSL modems, PDAs and other embedded systems), the combination of wildcard routing entries and redirect messages can be used to provide a simple routing-management scheme without the use of a higher-level policy process. Current connections can be rerouted after notification of the protocols by the protocols' pr_ctlinput() entries. Statistics are kept by the routing-table routines on the use of routing-redirect messages and on the latter's effect on the routing tables. A redirect causes the gateway for a route to be changed if the redirect applies to all destinations to which the route applies; otherwise, a new host route is cloned from the appropriate network route. Cloned host routes can easily pollute the routing table because an automated method for removing stale host routes created by redirects does not exist in the system. A user-level routing daemon will normally clean up stale host routes, but most hosts do not run routing daemons.

Routing-Table Interface

A protocol accesses the routing tables through three routines: one to allocate a route, one to free a route, and one to process a routing-redirect control message. The routine rtalloc() allocates a route; it is called with a pointer to a route structure, which contains the desired destination, as shown in Figure 12.10, and a pointer that will be set to reference the routing entry that is the best match for the destination. The destination is recorded so that subsequent output operations can check whether the new destination is the same as the previous one, allowing the same route to be used. The route returned is assumed to be held by the caller until released with a call to the RTFREE macro. All accesses to the routing table must be properly locked in FreeBSD and the RTFREE macro handles the locking as well as decrementing the route's reference count, freeing the route entry when the reference count reaches zero. TCP uses a host cache to hold onto routes for the duration of the connection's lifetime. Connectionless protocols, such as UDP, allocate and free routes whenever the routes' destination address changes. The rtalloc() routine simply checks whether the route already contains a reference to a valid route. If no route is referenced or the route is no longer valid, rtalloc() calls the rtalloc1 () routine to lookup a routing entry for the destination, passing a flag indicating whether the route will be used or is simply being checked. If packets will be sent, the route is created by cloning if necessary.

Figure 12.10. Data structures used in route allocation.

[View full size image]

The rtredirect() routine is called to process a redirect control message. It is called with a destination address and mask, the new gateway to that destination, and the source of the redirect. Redirects are accepted from only the current router for the destination. If a nonwildcard route exists to the destination, the gateway entry in the route is modified to point at the new gateway supplied. Otherwise, a new host route is cloned from an existing network route in the routing table. Routes to interfaces and routes to gateways that are not directly accessible from the host are ignored.

User-Level Routing Policies

The kernel routing facilities deliberately refrain from making policy decisions. Instead, routing policies are determined by user processes, which then add, delete, or change entries in the kernel routing tables. The decision to place policy decisions in a user process implies that routing-table updates may lag a bit behind the identification of new routes or the failure of existing routes. This period of instability is normally short if the routing process is implemented properly. Internet-specific advisory information, such as ICMP error messages, may also be read from raw sockets (described in Section 12.7).

Several routing-policy processes have been implemented. The system standard routing daemon, routed, uses the Routing Information Protocol (RIP) [Hedrick, 1988]. Many sites that require the use of other routing protocols or more configuration options than are provided by routed use either a commercial package or the open source Quagga Routing Suite [Ishiguro, 2003].

User-Level Routing Interface: Routing Socket

User-level processes that implement routing policy and protocols require an interface to the kernel routing table so that they can add, delete, and change kernel routes.

The interface to the kernel routing layer in FreeBSD uses a socket to communicate with the kernel routing layer. A privileged process creates a raw socket in the routing protocol family, AF_ROUTE, and then passes messages to and from the kernel routing layer. This socket operates like a normal datagram socket, including queueing of messages received at the socket, except that communication takes place between a user process and the kernel. Messages include a header with a message type identifying the action, as listed in Table 12.7. Messages to the kernel are requests to add, modify, or delete a route, or are requests for information about the route to a specific destination. The kernel sends a message in reply with the original request, an indication that the message is a reply, and an error number in case of failure. Because routing sockets are raw sockets, each open routing socket receives a copy of the reply and must filter for the messages it wants. The message header includes a process ID and a sequence number so that each process can determine whether this message is a reply to its own request and can match replies with requests. The kernel also sends messages as indications of asynchronous events, such as redirects and changes in local interface state. These messages allow a daemon to monitor changes in the routing table made by other processes, events detected by the kernel, and changes to the local interface addresses and state. The routing socket is also used to deliver requests for external resolution of a link-layer route when the RTF_XRESOLVE flag is set on a route entry.

Table 12.7. Routing message types.

Message type

Description

RTM_ADD

add route

RTM_DELETE

delete route

RTM_CHANGE

change metrics or flags

RTM_GET

report route and metrics

RTM_LOSING

kernel suspects partitioning

RTM_REDIRECT

told to use different route

RTM_MISS

lookup failed on this address

RTM_LOCK

lock specified metrics

RTM_OLDADD

caused by SIOCADDRT

RTM_OLDDEL

caused by SIOCDELRT

RTM_RESOLVE

request to resolve link address

RTM_NEWADDR

address added to interface

RTM_DELADDR

address removed from interface

RTM_IFINFO

interface going up or down

RTM_IFANNOUNCE

interface arrival and departure

RTM_NEWMADDR

multicast group membership being added to interface

RTM_DELMADDR

multicast group membership being deleted from interface

Requests to add or change a route include all the information needed for the route. The header has a field for the route flags listed in Table 12.6, and contains a rt_metrics structure of metrics that may be set or locked. The only metric that may be set is the MTU. All other metrics are ignored but were kept for backward compatibility with software that used the routing socket interface prior to FreeBSD 5.2. The header also carries a bit vector that describes the set of addresses carried in the message; the addresses follow the header as an array of variable-sized sockaddr structures. A destination address is required, as is a mask for network routes. A gateway address is generally required as well. The system normally determines the interface to be used by the route from the gateway address, using the interface shared with that gateway. By convention, direct routes contain the local address of the interface to be used. In some cases, the gateway address is not sufficient to determine the interface, and an interface address can be passed as well, generally using a sockaddr_dl structure containing the interface name or index (see Section 12.1).


12.6. Buffering and Congestion Control

A major factor affecting the performance of a protocol is the buffering policy. Lack of a proper buffering policy can force packets to be dropped, cause false windowing information to be emitted by protocols, fragment memory, and degrade the overall host performance. Because of these problems, most systems allocate a fixed pool of memory to the networking system and impose a policy optimized for normal network operation.

The FreeBSD networking system is not dramatically different in this respect. At boot time, a fixed amount of memory is allocated by the networking system for mbufs and mbuf clusters. At later times, more system memory may be requested for mbuf clusters as the need arises, up to a preconfigured limit; at no time, however, is this memory ever returned to the system. It would be possible to reclaim memory from network buffers but in the environments where the system has been used, storage for network packets has not been an issue, and thus storage reclamation has been left unimplemented.

Protocol Buffering Policies

When a socket is created, the protocol reserves some amount of buffer space for send and receive queues. These amounts define the high watermarks used by the socket routines in deciding when to block and unblock a process. The reservation of space does not currently result in any action by the memory-management routines.

Protocols that provide connection-level flow control base their decisions on the amount of space in the associated socket queues. That is, windows sent to peers are calculated based on the amount of free space in the socket's receive queue, whereas utilization of the send window received from a peer is dependent on the high watermark of the send queue.

Queue Limiting

Incoming packets from the network are always received unless memory allocation fails. However, each network-layer protocol input queue has an upper bound on the queue's length, and any packets exceeding that bound are discarded. It is possible for a host to be overwhelmed by excessive network traffic (e.g., if the host is acting as a router that connects a high-bandwidth network to a low-bandwidth network). As a defense mechanism, the queue limits can be adjusted to throttle network-traffic load on the system. Discarding packets is not always a satisfactory solution to this problem (simply dropping packets is likely to increase the load on a network); the queue lengths were incorporated mainly as a safeguard mechanism. On the other hand, limiting output queue lengths can be valuable on hosts that route traffic from a high-bandwidth network to a low-bandwidth network. The queue limit should be sufficiently high that transient overload can be handled by buffering, but allowing the queue to be too large causes network delays to increase to unacceptable levels.


12.7. Raw Sockets

A raw socket allows privileged users direct access to a protocol other than those normally used for transport of user data—for example, network-level protocols. Raw sockets are intended for knowledgeable processes that wish to take advantage of some protocol feature not directly accessible through the normal interface or for the development of protocols built atop existing protocols. For example, the ping program is implemented using a raw ICMP socket (see Section 13.8). The raw IP socket interface attempts to provide an identical interface to the one a protocol would have if it were resident in the kernel.

The raw socket support is built around a generic raw socket interface, possibly augmented by protocol-specific processing routines. This section describes only the core of the raw socket interface; details specific to particular protocols are not discussed. Some protocol families (including IPv4) use private versions of the routines and data structures described here.

Control Blocks

Every raw socket has a protocol control block of the form shown in Figure 12.11. Raw control blocks are kept on a singly linked list for performing lookups during packet dispatch. Associations may be recorded in fields referenced by the control block and may be used by the output routine in preparing packets for transmission. The rcb_proto field contains the protocol family and protocol number with which the raw socket is associated. The protocol, family, and addresses are used to filter packets on input, as described in the next subsection.

Figure 12.11. Raw-socket control block.

[View full size image]

A raw socket is datagram oriented: Each send or receive on the socket requires a destination address. Destination addresses may be supplied by the user or referenced via pointers to sockaddr structures in the control block and automatically installed in the outgoing packet by the output routine. If routing is necessary, it must be performed by an underlying protocol.

Input Processing

Input packets are assigned to raw sockets based on a simple pattern-matching scheme. Each protocol (and potentially some network interfaces) gives unassigned packets to the raw input routine with the call

void raw_input(
    struct mbuf *msg,
    struct sockproto *proto,
    struct sockaddr *src,
    struct sockaddr *dst);

Input packets are placed into the input queues of all raw sockets that match the header according to the following rules:

  1. The protocol family of the socket and header agree.

  2. If the protocol number in the socket is nonzero, then it agrees with that found in the packet header.

  3. If a local address is defined for the socket, the address format of the socket's local address is the same as the packet's destination address, and the two addresses agree exactly.

  4. Rule 3 is applied to the socket's foreign address and the packet's source address.

A basic assumption in the pattern-matching scheme is that addresses present in the control block and packet header (as constructed by the network interface and any raw input-protocol module) are in a canonical form that can be compared on a bit-for-bit basis. If multiple sockets match the incoming packet, the packet is copied as needed.

Output Processing

On output, each send request results in a call to the raw socket's raw_usend routine, which calls an output routine specific to the protocol or protocol family. Any necessary processing is done before the packet is delivered to the appropriate network interface.


12.8. Additional Network-Subsystem Topics

In this section, we discuss several aspects of the network subsystem that are not easy to categorize.

Out-of-Band Data

The ability to process out-of-band data is a facility specific to the stream-socket and sequenced-packet-socket abstractions. Little agreement appears to exist on what out-of-band data's semantics should be. TCP defines a notion called urgent data, in which in-line data are marked for urgent delivery. The protocol provides a mark on the data stream delimiting urgent data from subsequent normal data. The ISO/OSI protocols [Burruss, 1980] and numerous other protocols provide a fully independent logical transmission channel along which out-of-band data are sent. In addition, the number of data that can be sent in an out-of-band message varies from protocol to protocol, from 1 bit to 512 bytes or more.

A stream socket's notion of out-of-band data has been defined as the lowest reasonable common denominator. Out-of-band data are expected to be transmitted out of the normal sequencing and flow-control constraints of the data stream. A minimum of 1 byte of out-of-band data and one outstanding out-of-band message is expected to be provided by protocols supporting out-of-band messages. It is a protocol's prerogative to support larger-sized messages or more than one outstanding out-of-band message at a time.

Out-of-band data may be maintained by the protocol, stored separately from the socket's receive queue. They may also be prepended to the normal receive queue marked as out-of-band data. A socket-level option, SO_OOBINLINE, is provided to force all out-of-band data to be placed in the normal receive queue when urgent data are received. This option is provided because the 4.2BSD TCP implementation removed 1 byte of data from the data stream at the urgent mark for separate presentation. However, this removal caused problems when additional urgent data were sent before the first such byte was received by the application.

Placement of out-of-band data in the normal data stream can permit a protocol to hold several out-of-band messages simultaneously. This mechanism can avoid the loss of out-of-band messages caused by a process that responds slowly.

Address Resolution Protocol

The address resolution protocol (ARP) is a link-level protocol that provides a dynamic address-translation mechanism for networks that support broadcast or multicast communication [Plummer, 1982]. ARP maps 32-bit IPv4 addresses to 48-bit Ethernet addresses. Although ARP is not specific either to IPv4 protocol addresses or to Ethernet, the FreeBSD network subsystem supports only that combination, although it makes provision for additional combinations to be added. ARP is incorporated into the network-interface layer, although it logically sits between the network and network-interface layers.

The general idea of ARP is simple. A set of translations from network addresses to link-layer addresses is maintained. When an address-translation request is made to the ARP service by a network interface and the requested address is not in ARP's set of known translations, an ARP message is created that specifies the requested network address and an unknown link-layer address. This message is then broadcast by the interface with the expectation that a host attached to the network will know the translation—usually because the host is the intended target of the original message. If a response is received in a timely fashion, the ARP service uses the response to update its translation tables and to resolve the pending request, and the requesting network interface is then called to transmit the original message.

In practice, the simplicity of this algorithm is complicated by the necessity to avoid stale translation data, to minimize broadcasts when a target host is down, and to deal with failed translation requests. In addition, it is necessary to deal with packets for which transmission is attempted before the translation is completed. The ARP translation tables are implemented as a part of the routing table. The route to a local Ethernet is set up as a cloning route so that individual host routes will be created for each local host when referenced. When the route is cloned, the link layer creates an empty ARP entry associated with the route. The network output routine normally requires a routing lookup or a cached route, and it now passes a reference to the route to the interface output function.

A request is made to resolve an IPv4 address to an Ethernet address for an outgoing message by the call

int arpresolve(
    struct ifnet *ifp,
    struct rtentry *rt,
    struct mbuf *msg,
    struct sockaddr *dst,
    u_char *desten,
    struct rtentry *rt0);

ARP first checks its tables to see whether the destination address is a broadcast or multicast address, in which cases the Ethernet address can be calculated directly. Otherwise, it checks whether the route entry that was passed already contains a complete translation that has not timed out. If so, the gateway value in the route entry is a link-layer address for the destination, and its value is returned in desten for use as the destination address of the outgoing packet. If the link-layer address is not known or has timed out, ARP must queue the outgoing message for future transmission, and must broadcast a message requesting the Internet address translation. The time is recorded in the ARP entry when a request is broadcast, and no further broadcasts are made within the same second if additional transmissions are attempted. If another translation request is made before a reply is received, the queued message is discarded and only the newer one is saved. After some number of broadcasts without a reply (normally 5, in no less than 5 seconds), the route is changed to a reject route with an expiration time after 20 seconds, causing host-down errors to be returned in response to attempts to reach the host within that time.

At a later time—preferably before the timer has expired on the queued message—ARP will receive a response to its translation request. The received message is processed by the ether_demux.() routine, as invoked from the network thread. Because the packet has a packet type of ARP, it is enqueued for the arpintr() routine, analogous to other network-level protocol input routines. The ARP packet is processed to locate the translation entry in the routing table. If the message completes a pending translation, the entry is updated and the original message is passed back to the network interface for transmission. This time, the resultant call to arpresolve() will succeed without delay.

ARP input handling must cope with requests for the host's own address, as well as responses to translation requests that the host generated. The input module also watches for responses from other hosts that advertise a translation for its own IPv4 address. This monitoring is done to ensure that no two hosts on the same network believe that they have the same address (although this error may be detected, ARP's only recourse is to log a diagnostic message).

ARP normally times out completed translation entries in its cache after 20 minutes and incomplete translation entries after about 5 seconds. Entries may be marked permanent, however, in which case they are never removed. Entries may also be marked published, allowing one host to act as a surrogate for other hosts that do not support ARP or to act as a proxy for a host that is not on the Ethernet but is reached via a router.


Exercises

12.1 Name two key data structures used in the networking subsystem that are important in ensuring that the socket-layer software is kept independent of the networking implementation.

12.2 What work is done by the lower half of a network device driver? What work is done by the upper half of a network device driver?

12.3 Which routines in the protocol switch are called by the socket layer? Explain why each of these routines is called.

12.4 Assume that a reliably-delivered-message socket (SOCK_RDM) is a connectionless socket that guarantees reliable delivery of data and that preserves message boundaries. Which flags would a protocol that supported this type of socket have set in the pr_flags field of its protocol-switch entry?

12.5 Give an example of a network interface that is useful without an underlying hardware device.

12.6 Give two reasons why the addresses of a network interface are not in the network-interface data structure.

12.7 Why is the name or address of a socket kept at the network layer rather than at the socket layer?

12.8 Why does FreeBSD not attempt to enforce a rigid protocol-protocol interface structure?

12.9 Describe two tasks performed by a network-interface output routine.

12.10 Why is the identity of the network interface on which each message is received passed upward with the message?

12.11 Which routing policies are implemented in the kernel?

12.12 Describe three types of routes that can be found in the routing table that differ by the type of destination to which they apply.

12.13 What routing facility is designed mainly to support workstations?

12.14 What is a routing redirect? For what is it used?

12.15 Why do the output-packet queues for each network interface have limits on the number of packets that may be queued?

12.16 What does the SO_OOBINLINE socket option do? Why does it exist?

*12.17 Explain why it is impossible to use the raw socket interface to support parallel protocol implementations—some in the kernel and some in user mode. What modifications to the system would be necessary to support this facility?

*12.18 Previous versions of the system used a hashed routing lookup for a destination as a host or as a network. Name two ways in which the radix search algorithm in FreeBSD is more capable.


References

BBN, 1978. BBN, "Specification for the Interconnection of Host and IMP," Technical Report 1822, Bolt, Beranek, and Newman, Cambridge, MA, May 1978.

Burruss, 1980. J. Burruss, "Features of the Transport and Session Protocols," Report No. ICST/HLNP-80-1, National Bureau of Standards, Washington, DC, March 1980.

Hedrick, 1988. C. Hedrick, "Routing Information Protocol," RFC 1058, available from http://www.faqs.org/rfcs/rfcl058.html, June 1988.

Ishiguro, 2003. K. Ishiguro, Quagga, available from www.quagga.net, August 2003.

ISO, 1984. ISO, "Open Systems Interconnection: Basic Reference Model," ISO 7498, International Organization for Standardization, available from the American National Standards Institute, 1430 Broadway, New York, NY 10018, 1984.

Mogul & Deering, 1990. J. Mogul & S. Deering, "Path MTU Discovery," RFC 1191, available from http://www.faqs.org/rfcs/rfcll91.html, November 1990.

Plummer, 1982. D. Plummer, "An Ethernet Address Resolution Protocol," RFC 826, available from http://www.faqs.org/rfcs/rfc826.html, November 1982.

Postel, 1981. J. Postel, "Internet Control Message Protocol," RFC 792, available from http://www.faqs.org/rfcs/rfc792.html, September 1981.

Ritchie, 1984. D. Ritchie, "A Stream Input-Output System," AT&T Bell Laboratories Technical Journal, vol. 63, no. 8-2, pp. 1897-1910, October 1984.

Sedgewick, 1990. R. Sedgewick, Algorithms in C, Addison-Wesley, Reading, MA, 1990.

Sklower, 1991. K. Sklower, "A Tree-Based Packet Routing Table for Berkeley UNIX," USENIX Association Conference Proceedings, pp. 93-99, January 1991.

Wright & Stevens, 1995. G. R. Wright & W. R. Stevens, TCP/IP Illustrated, Volume 2, The Implementation, Addison-Wesley, Reading, MA, 1995.


Chapter 13. Network Protocols

Chapter 12 presented the network-communications architecture of FreeBSD. In this chapter, we examine the network protocols implemented within this framework. The FreeBSD system supports several major communication domains including IPv4, IPv6, Xerox Network Systems (NS), ISO/OSI, and the local domain (formerly known as the UNIX domain). The local domain does not include network protocols because it operates entirely within a single system. The IPv4 protocol suite was the first set of protocols implemented within the network architecture of 4.2BSD. Following the release of 4.2BSD, several proprietary protocol families were implemented by vendors within the network architecture. However, it was not until the addition of the Xerox NS protocols in 4.3BSD that the system's ability to support multiple network-protocol families was visibly demonstrated. Although some parts of the protocol interface were previously unused and thus unimplemented, the changes required to add a second network-protocol family did not substantially modify the network architecture. The implementation of the ISO/OSI networking protocols, as well as other changing requirements, led to a further refinement of the network architecture in 4.4BSD. Two new protocols that were added to the system, IPv6 and IPSec, required several changes because of its need to coexist simultaneously with IPv4. Those changes, as well as IPv6 and IPSec, are presented at the end of this chapter.

In this chapter, we concentrate on the organization and implementation of the IPv4 protocols. This protocol implementation is the standard on which the current Internet is built because it was publicly available when many vendors were looking for tuned and reliable communication protocols. Throughout this chapter we use Internet and IPv4 interchangeably. Specific mention will be made when talking about the new IPv6 protocols, which are meant to eventually supplant IPv4. After describing the overall architecture of the IPv4 protocols, we shall examine their operation according to the structure defined in Chapter 12. We shall also describe the significant algorithms used by the protocols within IPv4. We then shall discuss changes that the developers made in the system motivated by aspects of the IPv6 protocols and their implementation.


13.1. IPv4 Network Protocols

IPv4 was developed under the sponsorship of DARPA, for use on the ARPANET [DARPA, 1983; McQuillan & Walden, 1977]. The protocols are commonly known as TCP/IP, although TCP and IP are only two of the many protocols in the family. These protocols do not assume a reliable subnetwork that ensures delivery of data. Instead, IPv4 was devised for a model in which hosts were connected to networks with varying characteristics and the networks were interconnected by routers. The Internet protocols were designed for packet-switching networks using datagrams sent over links such as Ethernet that provide no indication of delivery.

This model leads to the use of at least two protocol layers. One layer operates end to end between two hosts involved in a conversation. It is based on a lower-level protocol that operates on a hop-by-hop basis, forwarding each message through intermediate routers to the destination host. In general, there exists at least one protocol layer above the other two: It is the application layer. The three layers correspond roughly to levels 3 (network), 4 (transport), and 7 (application) in the ISO Open Systems Interconnection reference model [ISO, 1984].

The protocols that support this model have the layering illustrated in Figure 13.1. The Internet Protocol (IP) is the lowest-level protocol in the Model; this level corresponds to the ISO network layer. IP operates hop-by-hop as a datagram is sent from the originating host to the destination via any intermediate routers. It provides the network-level services of host addressing, routing, and, if necessary, packet fragmentation and reassembly if intervening networks cannot send an entire packet in one piece. All the other protocols use the services of IP. The Transmission Control Protocol (TCP) and User Datagram Protocol (UDP) are transport-level protocols that provide additional facilities to applications that use IP. Each protocol adds a port identifier to IP's host address so that local and remote sockets can be identified. TCP provides connection-oriented, reliable, unduplicated, and flow-controlled transmission of data; it supports the stream socket type in the Internet domain. UDP provides a data checksum for checking integrity in addition to a port identifier, but otherwise adds little to the services provided by IP. UDP is the protocol used by datagram sockets in the Internet domain. The Internet Control Message Protocol (ICMP) is used for error reporting and for other, simple network-management tasks; it is logically a part of IP but, like the transport protocols, is layered above IP. It is usually not accessed by users. Raw access to the IP and ICMP protocols is possible through raw sockets (see Section 12.7 for information on this facility).

Figure 13.1. IPv4 protocol layering. Key: TCP—Transmission Control Protocol; UDP—User Datagram Protocol; IP—Internet Protocol; ICMP—Internet Control Message Protocol.

The Internet protocols were designed to support heterogeneous host systems and architectures that use a wide variety of internal data representations. Even the basic unit of data, the byte, was not the same on all host systems; one common type of host supported variable-sized bytes. The network protocols, however, require a standard representation. This representation is expressed using the octet—an 8-bit byte. We shall use this term as it is used in the protocol specifications to describe network data, although we continue to use the term byte to refer to data or storage within the system. All fields in the Internet protocols that are larger than an octet are expressed in network byte order, with the most significant octet first. The FreeBSD network implementation uses a set of routines or macros to convert 16-bit and 32-bit integer fields between host and network byte order on hosts (such as PC systems) that have a different native ordering.

IPv4 Addresses

An IPv4 address is a 32-bit number that identifies the network on which a host resides as well as uniquely identifying a network interface on that host. It follows that a host with network interfaces attached to multiple networks has multiple addresses. Network addresses are assigned in blocks by Regional Internet Registries (RIRs) to Internet Service Providers (ISPs), which then dole out addresses to companies or individual users. If address assignment were not done in this centralized way, conflicting addresses could arise in the network, and it would be impossible to route packets correctly.

Historically IPv4 addresses were rigidly divided into three classes (A, B, and C) to address the needs of large, medium, and small networks [Postel, 1981a]. Three classes proved to be too restrictive and also too wasteful of address space. The current IPv4 addressing scheme is called Classless Inter-Domain Routing (CIDR) [Fuller et al., 1993]. In the CIDR scheme each organization is given a contiguous group of addresses described by a single value and a netmask. For example, an ISP might have a group of addresses defined by an 18-bit netmask. This means that the network is defined by the first 18 bits, and the remaining 14 bits can potentially be used to identify hosts in the network. In practice, the number of hosts is less because the ISP will further break up this space into smaller networks, which will reduce the number of bits that can effectively be used. It is because of this scheme that routing entries store arbitrary netmasks with routes.

Each Internet address assigned to a network interface is maintained in an in_ifaddr structure that contains a protocol-independent interface-address structure and additional information for use in the Internet domain (see Figure 13.2). When an interface's network mask is specified, it is recorded in the ia_subnetmask field of the address structure. The network mask, ia_netmask, is still calculated based on the type of the network number (class A, B, or C) when the interface's address is assigned, but this is no longer used to determine whether a destination is on or off the local subnet. The system interprets local Internet addresses using ia_subnetmask value. An address is considered to be local to the subnet if the field under the subnetwork mask matches the subnetwork field of an interface address.

Figure 13.2. Internet interface address structure (in_jfaddr).

Broadcast Addresses

On networks capable of supporting broadcast datagrams, 4.2BSD used the address with a host part of zero for broadcasts. After 4.2BSD was released, the Internet broadcast address was defined as the address with a host part of all is [Mogul, 1984]. This change and the introduction of subnets both complicated the recognition of broadcast addresses. Hosts may use a host part of 0s or 1s to signify broadcast, and some may understand the presence of subnets, whereas others may not. For these reasons, 4.3BSD and later BSD systems set the broadcast address for each interface to be the host value of all is but allow the alternate address to be set for backward compatibility. If the network is subnetted, the subnet field of the broadcast address contains the normal subnet number. The logical broadcast address for the network also is calculated when the address is set; this address would be the standard broadcast address if subnets were not in use. This address is needed by the IP input routine to filter input packets. On input, FreeBSD recognizes and accepts subnet and network broadcast addresses with host parts of 0s or 1s, as well as the address with 32 bits of 1 ("broadcast on this physical link").

Internet Multicast

Many link-layer networks, such as Ethernet, provide a multicast capability that can address groups of hosts but is more selective than broadcast because it provides several different multicast group addresses. IP provides a similar facility at the network-protocol level, using link-layer multicast where available [Deering, 1989]. IP multicasts are sent using destination addresses with high-order bits set to 1110. Unlike host addresses, multicast addresses do not contain network and host portions; instead, the entire address names a group, such as a group of hosts using a particular service. These groups can be created dynamically, and the members of the group can change over time. IP multicast addresses map directly to physical multicast addresses on networks such as the Ethernet, using the low 24 bits of the IP address along with a constant 24-bit prefix to form a 48-bit link-layer address.

For a socket to use multicast, it must join a multicast group using the setsockopt system call. This call informs the link layer that it should receive multicasts for the corresponding link-layer address, and it also sends a multicast membership report using the Internet Group Management Protocol (Cain et al., 2002) . Multicast agents on the network can thus keep track of the members of each group. Multicast agents receive all multicast packets from directly attached networks and forward multicast datagrams as needed to group members on other networks. This function is similar to the role of routers that forward normal (unicast) packets, but the criteria for packet forwarding are different, and a packet can be forwarded to multiple neighboring networks.

Internet Ports and Associations

At the IP level, packets are addressed to a host rather than to a process or communications port. However, each packet contains an 8-bit protocol number that identifies the next protocol that should receive the packet. Internet transport protocols use an additional identifier to designate the connection or communications port on the host. Most protocols (including TCP and UDP) use a 16-bit port number for this purpose. Each transport protocol maintains its own mapping of port numbers to processes or descriptors. Thus, an association, such as a connection, is fully specified by the tuple <source address, destination address, protocol number, source port, destination port>. Connection-oriented protocols, such as TCP, must enforce the uniqueness of associations; other protocols generally do so as well. When the local part of the address is set before the remote part, it is necessary to choose a unique port number to prevent collisions when the remote part is specified.

Protocol Control Blocks

For each TCP- or UDP-based socket, an Internet protocol control block (an inpcb structure) is created to hold Internet network addresses, port numbers, routing information, and pointers to any auxiliary data structures. TCP, in addition, creates a TCP control block (a tcpcb structure) to hold the wealth of protocol state information necessary for its implementation. Internet control blocks for use with TCP are maintained on a doubly linked list private to the TCP protocol module. Internet control blocks for use with UDP are kept on a similar list private to the UDP protocol module. Two separate lists are needed because each protocol in the Internet domain has a distinct space of port identifiers. Common routines are used by the individual protocols to add new control blocks to a list, fix the local and remote parts of an association, locate a control block by association, and delete control blocks. IP demultiplexes message traffic based on the protocol identifier specified in its protocol header, and each higher-level protocol is then responsible for checking its list of Internet control blocks to direct a message to the appropriate socket. Figure 13.3 shows the linkage between the socket data structure and these protocol-specific data structures.

Figure 13.3. Internet protocol data structures.

[View full size image]

The implementation of the Internet protocols is tightly coupled, as befits the strong intertwining of the protocols. For example, the transport protocols send and receive packets including not only their own header, but also an IP pseudo-header containing the source and destination address, the protocol identifier, and a packet length. This pseudo-header is included in the transport-level packet checksum.

We are now ready to examine the operation of the Internet protocols. We begin with UDP, because it is far simpler than TCP.


13.2. User Datagram Protocol (UDP)

The User Datagram Protocol (UDP) [Postel, 1980] is a simple, unreliable datagram protocol that provides only peer-to-peer addressing and optional data checksums. In FreeBSD, checksums are enabled or disabled on a systemwide basis and cannot be enabled or disabled on individual sockets. UDP protocol headers are extremely simple, containing only the source and destination port numbers, the datagram length, and the data checksum. The host addresses for a datagram are provided by the IP pseudo-header.

Initialization

When a new datagram socket is created in the Internet domain, the socket layer locates the protocol-switch entry for UDP and calls the udp_attach() routine with the socket as a parameter. UDP uses in_pcballoc() to create a new protocol control block on its list of current sockets. It also sets the default limits for the socket send and receive buffers. Although datagrams are never placed in the send buffer, the limit is set as an upper limit on datagram size; the UDP protocol-switch entry contains the flag PR_ATOMIC, requiring that all data in a send operation be presented to the protocol at one time.

If the application program wishes to bind a port number—for example, the well-known port for some datagram service—it calls the bind system call. This request reaches UDP as a call to the udp_bind() routine. The binding may also specify a specific host address, which must be an address of an interface on this host. Otherwise, the address will be left unspecified, matching any local address on input, and with an address chosen as appropriate on each output operation. The binding is done by in_pcbbind(), which verifies that the chosen port number (or address and port) is not in use and then records the local part of the association.

To send datagrams, the system must know the remote part of an association. A program can specify this address and port with each send operation using sendto or sendmsg, or it can do the specification ahead of time with the connect system call. In either case, UDP uses the in_pcbconnect() function to record the destination address and port. If the local address was not bound, and if a route for the destination is found, the address of the outgoing interface is used as the local address. If no local port number was bound, one is chosen at this time.

Output

A system call that sends data reaches UDP as a call to the udp_send() routine that takes a chain of mbufs containing the data for the datagram. If the call provided a destination address, the address is passed as well; otherwise, the address from a prior connect call is used. The actual output operation is done by udp_output(),

static int udp_output(
    struct inpcb *inp,
    struct mbuf *msg,
    struct mbuf *addr,
    struct mbuf *control,
    struct thread *td);

where inp is an IPv4 protocol control block, msg is chain of mbufs that contain the data to be sent, addr is an optional mbuf containing the destination address, and td is a pointer to a thread structure. Thread structures were discussed in Section 4.2 and are used within the network stack to identify the sender of a packet, which is why they are only used with output routines. Any ancillary data in control are discarded. The destination address could have been prespecified with a connect call; otherwise, it must be provided in the send call. UDP simply prepends its own header, fills in the UDP header fields and those of a prototype IP header, and calculates a checksum before passing the packet on to the IP module for output:

int ip_output(
    struct mbuf *msg,
    struct mbuf *opt,
    struct route *ro,
    int flags,
    struct ip_moptions *imo,
    struct inpcb *inp);

The call to IP's output routine is more complicated than UDP's because the IP routine needs to have more information specified about the endpoint to which it is communicating. The msg parameter indicates the message to be sent, and the opt parameter may specify a list of IP options that should be placed in the IP packet header. For multicast destinations, the into parameter may reference multicast options, such as the choice of interface and hop count for multicast packets. IP options may be set for a socket with the setsockopt system call specifying the IP protocol level and option IP_OPTIONS. These options are stored in a separate mbuf, and a pointer to this mbuf is stored in the protocol control block for a socket. The pointer to the options is passed to ip_output() with each packet sent. The ro parameter is optional and is passed as NULL by the udp_output() routine so that IP will determine a route for the packet. The flags parameter indicates whether the user is allowed to transmit a broadcast message and whether routing is to be bypassed for the message being sent (see Section 13.3). The broadcast flag may be inconsequential if the underlying hardware does not support broadcast transmissions. The flags also indicate whether the packet includes an IP pseudo-header or a completely initialized IP header, as when IP forwards packets.

Input

All Internet transport protocols that are layered directly on top of IP use the following calling convention when receiving packets from IP:

(void) (*pr_input) (
    struct mbuf *m,
    int off);

Each mbuf chain passed is a single packet to be processed by the protocol module. The packet includes the IP header in lieu of a pseudo-header, and the IP header length is passed as an offset: the second parameter. The UDP input routine udp_input() is typical of protocol input routines. It first verifies that the length of the packet is at least as long as the IP plus UDP headers, and it uses m_pullup() to make the header contiguous. The udp_input() routine then checks that the packet is the correct length and calculates a checksum for the data in the packet. If any of these tests fail, the packet is discarded, and error counts are increased. Any multicasting issues are handled next. Finally, the protocol control block for the socket that is to receive the data is located by in_pcblookup() using the addresses and port numbers in the packet. There might be multiple control blocks with the same local port number but different local or remote addresses; if so, the control block with the best match is selected. An exact association matches best, but if none exists, a socket with the correct local port number but unspecified local address, remote port number, or remote address will match. A control block with unspecified local or remote addresses thus acts as a wildcard that receives packets for its port if no exact match is found. If a control block is located, the data and the address from which the packet was received are placed in the receive buffer of the indicated socket with udp_append(). If the destination address is a multicast address, copies of the packet are delivered to each socket with matching addresses. Otherwise, if no receiver is found and if the packet was not addressed to a broadcast or multicast address, an ICMP port unreachable error message is sent to the originator of the datagram. This error message normally has no effect, as the sender typically connects to this destination only temporarily, and destroys the association before new input is processed. However, if the sender still has a fully specified association, it may receive notification of the error.

Control Operations

UDP supports no control operations and passes calls to its pr_ctloutput() entry directly to IP. It has a simple pr_ctlinput() routine that receives notification of any asynchronous errors. Errors are passed to any datagram socket with the indicated destination; only sockets with a destination fixed by a connect call may be notified of errors asynchronously. Such errors are simply noted in the appropriate socket, and socket wakeups are issued if the process is selecting or sleeping while waiting for input.

When a UDP datagram socket is closed, the udp_detach() routine is called. The protocol control block and its contents are simply deleted with in_pcbdetach(); no other processing is required.


13.3. Internet Protocol (IP)

Having examined the operation of a simple transport protocol, we continue with a discussion of the network-layer protocol [Postel, 1981a; Postel et al., 1981]. The Internet Protocol (IP) is the level responsible for host-to-host addressing and routing, packet forwarding, and packet fragmentation and reassembly. Unlike the transport protocols, it does not always operate for a socket on the local host; it may forward packets, receive packets for which there is no local socket, or generate error packets in response to these situations.

The functions done by IP are illustrated by the contents of its packet header, shown in Figure 13.4. The header identifies source and destination hosts and the destination protocol, and it contains header and packet lengths. The identification and fragment fields are used when a packet or fragment must be broken into smaller sections for transmission on its next hop and to reassemble the fragments when they arrive at the destination. The fragmentation flags are Don't Fragment and More Fragments; the latter flag plus the offset are enough information to assemble the fragments of the original packet at the destination.

Figure 13.4. IPv4 header. IHL is the Internet header length specified in units of four octets. Options are delimited by IHL. All field lengths are given in bits.

[View full size image]

IP options are present in an IP packet if the header length field has a value larger than the minimum, which is 20 bytes. The no-operation option and the end-of-option-list option are each one octet in length. All other options are self-encoding, with a type and length preceding any additional data. Hosts and routers are thus able to skip over options that they do not implement. Examples of existing options are the timestamp and record-route options, which are updated by each router that forwards a packet, and the source-route options, which supply a complete or partial route to the destination.

In practice these are rarely used and most network operators silently drop packets with the source-route option because it makes it difficult to manage traffic on the network.

Output

We have already seen the calling convention for the IP output routine, which is

int ip_output(
    struct mbuf *msg,
    struct mbuf *opt,
    struct route *ro,
    int flags,
    struct ip_moptions *imo,
    struct inpcb *inp);

As described in the subsection on output in Section 13.2, the parameter msg is an mbuf chain containing the packet to be sent, including a skeletal IP header; opt is an optional mbuf containing IP options to be inserted after the header. If the route ro is given, it may contain a reference to a routing entry (rtentry structure), which specifies a route to the destination from a previous call, and in which any new route will be left for future use. Since cached routes were removed from the inpcb structure in FreeBSD 5.2, this cached route is seldomly used. The flags may allow the use of broadcast or may indicate that the routing tables should be bypassed. If present, imo includes options for multicast transmissions. The protocol control block, inp, is used by the IPSec subsystem (see Section 13.10) to hold data about security associations for the packet.

The outline of the work done by ip_output() is as follows:

  • Insert any IP options.

  • Fill in the remaining header fields (IP version, zero offset, header length, and a new packet identification) if the packet contains an IP pseudo-header.

  • Determine the route (i.e., outgoing interface and next-hop destination).

  • Check whether the destination is a multicast address. If it is, determine the outgoing interface and hop count.

  • Check whether the destination is a broadcast address; if it is, check whether broadcast is permitted.

  • Do any IPSec manipulations that are necessary on the packet such as encryption.

  • See if there are any filtering rules that would modify the packet or prevent us from sending it.

  • If the packet size is no larger than the maximum packet size for the outgoing interface, compute the checksum and call the interface output routine.

  • If the packet size is larger than the maximum packet size for the outgoing interface, break the packet into fragments and send each in turn.

We shall examine the routing step in more detail. First, if no route reference is passed as a parameter, an internal routing reference structure is used temporarily. A route structure that is passed from the caller is checked to see that it is a route to the same destination and that it is still valid. If either test fails, the old route is freed. After these checks, if there is no route, rtalloc_ign() is called to allocate a route. The route returned includes a pointer to the outgoing interface. The interface information includes the maximum packet size, flags including broadcast and multicast capability, and the output routine. If the route is marked with the RTF_GATEWAY flag, the address of the next-hop router is given by the route; otherwise, the packet's destination is the next-hop destination. If routing is to be bypassed because of a MSG_DONTROUTE option (see Section 11.1) or a SO_DONTROUTE option, a directly attached network shared with the destination is sought; if there is no directly attached network, an error is returned. Once the outgoing interface and next-hop destination are found, enough information is available to send the packet.

As described in Chapter 12, the interface output routine normally validates the destination address and places the packet on its output queue, returning errors only if the interface is down, the output queue is full, or the destination address is not understood.

Input

In Chapter 12, we described the reception of a packet by a network interface and the packet's placement on the input queue for the appropriate protocol. The network-interface handler then schedules the protocol to run by setting a corresponding bit in the network status word and scheduling the network thread. The IPv4 input routine is invoked via this software interrupt when network in`terfaces receive messages for the IPv4 protocol. The input routine, ip_input(), is called with an mbuf that contains the packet it is to process. The dequeueing of packets and the calls into the input routine are handled by the network thread calling netisr_dispatch(). A packet is processed in one of four ways: it is passed as input to a higher-level protocol, it encounters an error that is reported back to the source, it is dropped because of an error, or it is forwarded to the next hop on its path to its destination. In outline form, the steps in the processing of a packet on input are as follows:

1. Verify that the packet is at least as long as an IPv4 header and ensure that the header is contiguous.

2. Checksum the header of the packet and discard the packet if there is an error.

3. Verify that the packet is at least as long as the header indicates and drop the packet if it is not. Trim any padding from the end of the packet.

4. Do any filtering or security functions required by ipfw or IPSec.

5. Process any options in the header.

6. Check whether the packet is for this host. If it is, continue processing the packet. If it is not, and if acting as a router, try to forward the packet. Otherwise, drop the packet.

7. If the packet has been fragmented, keep it until all its fragments are received and reassembled, or until it is too old to keep.

8. Pass the packet to the input routine of the next-higher-level protocol.

When the incoming packet is passed into the input routine, it is accompanied by a pointer to the interface on which the packet was received. This information is passed to the next protocol, to the forwarding function, or to the error-reporting function. If any error is detected and is reported to the packet's originator, the source address of the error message will be set according to the packet's destination and the incoming interface.

The decision whether to accept a received packet for local processing by a higher-level protocol is not as simple as one might think. If a host has multiple addresses, the packet is accepted if its destination matches any one of those addresses. If any of the attached networks support broadcast and the destination is a broadcast address, the packet is also accepted.

The IPv4 input routine uses a simple and efficient scheme for locating the input routine for the receiving protocol of an incoming packet. The protocol field in the packet is 8 bits long; thus, there are 256 possible protocols. Fewer than 256 protocols are defined or implemented, and the Internet protocol switch has far fewer than 256 entries. Therefore, ip_input() uses a 256-element mapping array to map from the protocol number to the protocol-switch entry of the receiving protocol. Each entry in the array is initially set to the index of a raw IP entry in the protocol switch. Then, for each protocol with a separate implementation in the system, the corresponding map entry is set to the index of the protocol in the IP protocol switch. When a packet is received, IP simply uses the protocol field to index into the mapping array and calls the input routine of the appropriate protocol.

Forwarding

Implementations of IPv4 traditionally have been designed for use by either hosts or routers, rather than by both. That is, a system was either an endpoint for packets (as source or destination) or a router (which forwards packets between hosts on different networks but only uses upper-level protocols for maintenance functions). Traditional host systems do not incorporate packet-forwarding functions; instead, if they receive packets not addressed to them, they simply drop the packets. 4.2BSD was the first common implementation that attempted to provide both host and router services in normal operation. This approach had advantages and disadvantages. It meant that 4.2BSD hosts connected to multiple networks could serve as routers as well as hosts, reducing the requirement for dedicated router hardware. Early routers were expensive and not especially powerful. Alternatively, the existence of router-function support in ordinary hosts made it more likely for misconfiguration errors to result in problems on the attached networks. The most serious problem had to do with forwarding of a broadcast packet because of a misunderstanding by either the sender or the receiver of the packet's destination. The packet-forwarding router functions are disabled by default in FreeBSD. They may be enabled at run time with the sysctl call. Hosts not configured as routers never attempt to forward packets or to return error messages in response to misdirected packets. As a result, far fewer misconfiguration problems are capable of causing synchronized or repetitive broadcasts on a local network, called broadcast storms.

The procedure for forwarding IP packets received at a router but destined for another host is the following:

1. Check that forwarding is enabled. If it is not, drop the packet.

2. Check that the destination address is one that allows forwarding. Packets destined for network 0, network 127 (the official loopback network), or illegal network addresses cannot be forwarded.

3. Save at most 64 octets of the received message in case an error message must be generated in response.

4. Determine the route to be used in forwarding the packet.

5. If the outgoing route uses the same interface as that on which the packet was received, and if the originating host is on that network, send an ICMP redirect message to the originating host. (ICMP is described in Section 13.8.)

6. Handle any IPSec updates that must be made to the packet header.

7. Call ip_output() to send the packet to its destination or to the next-hop gateway.

8. If an error is detected, send an ICMP error message to the source host.

Multicast transmissions are handled separately from other packets. Systems may be configured as multicast routers independently from other routing functions. Multicast routers receive all incoming multicast packets, and forward those packets to local receivers and group members on other networks according to group memberships and the remaining hop count of incoming packets.


13.4. Transmission Control Protocol (TCP)

The most used protocol of the Internet protocol suite is the Transmission Control Protocol (TCP) [Cerf & Kahn, 1974; Postel, 1981b]. TCP is the reliable connection-oriented stream transport protocol on top of which most application protocols are layered. It includes several features not found in the other transport and network protocols described so far:

  • Explicit and acknowledged connection initiation and termination

  • Reliable, in-order, unduplicated delivery of data

  • Flow control

  • Out-of-band indication of urgent data

  • Congestion avoidance

Because of these features, the TCP implementation is much more complicated than are those of UDP and IP. These complications, along with the prevalence of the use of TCP, make the details of TCP's implementation both more critical and more interesting than are the implementations of the simpler protocols. Figure 13.5 shows the flow of data through a TCP connection. We shall begin with an examination of the TCP itself and then continue with a description of its implementation in FreeBSD.

Figure 13.5. Data flow through a TCP/IP connection over an Ethernet. Key: ETHER—Ethernet header; PIP—pseudo-IP header; IP—IP header; TCP—TCP header; IF—interface.

[View full size image]

A TCP connection may be viewed as a bidirectional, sequenced stream of data transferred between two peers. The data may be sent in packets of varying sizes and at varying intervals—for example, when they are used to support a login session over the network. The stream initiation and termination are explicit events at the start and end of the stream, and they occupy positions in the sequence space of the stream so that they can be acknowledged in the same way as data are. Sequence numbers are 32-bit numbers from a circular space; that is, comparisons are made modulo 232, so zero is the next sequence number after 232-l. The sequence numbers for each direction start with an arbitrary value, called the initial sequence number, sent in the initial packet for a connection. Following Bellovin [1996], the TCP implementation selects the initial sequence number by computing a function over the 4 tuple local port, foreign port, local address, foreign address that uniquely identifies the connection, and then adding a small offset based on the current time. This algorithm prevents the spoofing of TCP connections by an attacker guessing the next initial sequence number for a connection. This must be done while also guaranteeing that an old duplicate packet will not match the sequence space of a current connection.

Each packet of a TCP connection carries the sequence number of its first datum and (except during connection establishment) an acknowledgment of all contiguous data received. A TCP packet is known as a segment because it begins at a specific location in the sequence space and has a specific length. Acknowledgments are specified as the sequence number of the next sequence number not yet received. Acknowledgments are cumulative and thus may acknowledge data received in more than one (or part of one) packet. A packet may or may not contain data, but it always contains the sequence number of the next datum to be sent.

Flow control in TCP is done with a sliding-window scheme. Each packet with an acknowledgment contains a window, which is the number of octets of data that the receiver is prepared to accept, beginning with the sequence number in the acknowledgment. The window is a 16-bit field, limiting the window to 65535 octets by default; however, the use of a larger window may be negotiated. Urgent data are handled similarly; if the flag indicating urgent data is set, the urgent-data pointer is used as a positive offset from the sequence number of the packet to indicate the extent of urgent data. Thus, TCP can send notification of urgent data without sending all intervening data, even if the flow-control window would not allow the intervening data to be sent.

The complete header for a TCP packet is shown in Figure 13.6. The flags include SYN and FIN, denoting the initiation (synchronization) and completion of a connection. Each of these flags occupies a sequence space of one. A complete connection thus consists of a SYN, zero or more octets of data, and a FIN sent from each peer and acknowledged by the other peer. Additional flags indicate whether the acknowledgment field (ACK) and urgent fields (URG) are valid, and include a connection-abort signal (RST). Options are encoded in the same way as are IP options: The no-operation and end-of-options options are single octets, and all other options include a type and a length. The only option in the initial specification of TCP indicates the maximum segment (packet) size that a correspondent is willing to accept; this option is used only during initial connection establishment. Several other options have been defined. To avoid confusion, the protocol standard allows these options to be used in data packets only if both end-points include them during establishment of the connection.

Figure 13.6. TCP packet header.

[View full size image]

TCP Connection States

The connection-establishment and connection-completion mechanisms of TCP are designed for robustness. They serve to frame the data that are transferred during a connection so that not only the data but also their extent are communicated reliably. In addition, the procedure is designed to discover old connections that have not terminated correctly because of a crash of one peer or loss of network connectivity. If such a half-open connection is discovered, it is aborted. Hosts choose new initial sequence numbers for each connection to lessen the chances that an old packet may be confused with a current connection.

The normal connection-establishment procedure is known as a three-way handshake. Each peer sends a SYN to the other, and each in turn acknowledges the other's SYN with an ACK. In practice, a connection is normally initiated by a client attempting to connect to a server listening on a well-known port. The client chooses a port number and initial sequence number and uses these selections in the initial packet with a SYN. The server creates a protocol control block for the pending connection and sends a packet with its initial sequence number, a SYN, and an ACK of the client's SYN. The client responds with an ACK of the server's SYN, completing connection establishment. As the ACK of the first SYN is piggybacked on the second SYN, this procedure requires three packets, leading to the term three-way handshake. The protocol still operates correctly if both peers attempt to start a connection simultaneously, although the connection setup requires four packets.

FreeBSD includes three options along with the SYN when initiating a connection. One contains the maximum segment size that the system is willing to accept [Jacobson et al., 1992]. The second of these options specifies a window-scaling value expressed as a binary shift value, allowing the window to exceed 65535 octets. If both peers include this option during the three-way handshake, both scaling values take effect; otherwise, the window value remains in octets. The third option is a timestamp option. If this option is sent in both directions during connection establishment, it will also be sent in each packet during data transfer. The data field of the timestamp option includes a timestamp associated with the current sequence number and also echoes a timestamp associated with the current acknowledgment. Like the sequence space, the timestamp uses a 32-bit field and modular arithmetic. The unit of the timestamp field is not defined, although it must fall between 1 millisecond and 1 second. The value sent by each system must be monotonically nondecreasing during a connection. FreeBSD uses the value of ticks, which is incremented at the system clock rate, HZ. These time-stamps can be used to implement round-trip timing. They also serve as an extension of the sequence space to prevent old duplicate packets from being accepted; this extension is valuable when a large window or a fast path, such as an Ethernet, is used.

After a connection is established, each peer includes an acknowledgment and window information in each packet. Each may send data according to the window that it receives from its peer. As data are sent by one end, the window becomes filled. As data are received by the peer, acknowledgments may be sent so that the sender can discard the data from its send queue. If the receiver is prepared to accept additional data, perhaps because the receiving process has consumed the previous data, it will also advance the flow-control window. Data, acknowledgments, and window updates may all be combined in a single message.

If a sender does not receive an acknowledgment within some reasonable time, it retransmits data that it presumes were lost. Duplicate data are discarded by the receiver but are acknowledged again in case the retransmission was caused by loss of the acknowledgment. If the data are received out of order, the receiver generally retains the out-of-order data for use when the missing segment is received. Out-of-order data cannot be acknowledged because acknowledgments are cumulative. A selective acknowledgment mechanism was introduced in Jacobson et al. [1992] but is not implemented in FreeBSD.

Each peer may terminate data transmission at any time by sending a packet with the FIN bit. A FIN represents the end of the data (like an end-of-file indication). The FIN is acknowledged, advancing the sequence number by 1. The connection may continue to carry data in the other direction until a FIN is sent in that direction. The acknowledgment of the FIN terminates the connection. To guarantee synchronization at the conclusion of the connection, the peer sending the last ACK of a FIN must retain state long enough that any retransmitted FIN packets would have reached it or have been discarded; otherwise, if the ACK were lost and a retransmitted FIN were received, the receiver would be unable to repeat the acknowledgment. This interval is arbitrarily set to twice the maximum expected segment lifetime (known as 2MSL).

The TCP input-processing module and timer modules must maintain the state of a connection throughout that connection's lifetime. Thus, in addition to processing data received on the connection, the input module must process SYN and FIN flags and other state transitions. The list of states for one end of a TCP connection is given in Table 13.1. Figure 13.7 (on page 532) shows the finite-state machine made up by these states, the events that cause transitions, and the actions during the transitions.

Table 13.1. TCP connection states.

State

Description

States involved while a connection becomes established

CLOSED

closed

LISTEN

listening for connection

SYN SENT

active, have sent SYN

SYN RECEIVED

have sent and received SYN

State during an established connection

ESTABLISHED

established

States involved when the remote end initiates a connection shutdown

CLOSE WAIT

have received FIN, waiting for close

LAST ACK

have received FIN and close; awaiting FIN ACK

CLOSED

closed

States involved when the local end initiates a connection shutdown

FIN WAIT 1

have closed, sent FIN

CLOSING

closed, exchanged FIN; awaiting FIN ACK

FIN WAIT 2

have closed, FIN is acknowledged; awaiting FIN

TIME WAIT

in 2MSL[ [image]] quiet wait after close

CLOSED

closed

[ [image]] 2MSL — twice maximum segment lifetime.

Figure 13.7. TCP state diagram. Key: TCB—TCP control block; 2MSL—twice maximum segment lifetime.

If a connection is lost because of a crash or timeout on one peer but is still considered established by the other, then any data sent on the connection and received at the other end will cause the half-open connection to be discovered. When a half-open connection is detected, the receiving peer sends a packet with the RST flag and a sequence number derived from the incoming packet to signify that the connection is no longer in existence.

Sequence Variables

Each TCP connection maintains a large set of state variables in the TCP control block. This information includes the connection state, timers, options and state flags, a queue that holds data received out of order, and several sequence number variables. The sequence variables are used to define the send and receive sequence space, including the current window for each. The window is the range of data sequence numbers that are currently allowed to be sent, from the first octet of data not yet acknowledged up to the end of the range that has been offered in the window field of a header. The variables used to define the windows in FreeBSD are a superset of those used in the protocol specification [Postel, 1981b]. The send and receive windows are shown in Figure 13.8. The meanings of the sequence variables are listed in Table 13.2.

Table 13.2. TCP sequence variables.

Variable

Description

snd_una

lowest send sequence number not yet acknowledged

snd_nxt

next data sequence to be sent

snd_wnd

number of data octets peer will receive, starting with snd_una

snd_max

highest sequence number sent

rcv_nxt

next receive sequence number expected

rcv_wnd

number of octets past rcv_nxt that may be accepted

rcv_adv

last octet of receive window advertised to peer

ts_recent

most recent timestamp received from peer

ts_recent_age

time when ts_recent was received

Figure 13.8. TCP sequence space.

The area between snd_una and snd_una + snd_wnd is known as the send window. Data for the range snd_una to snd_max have been sent but not yet acknowledged and are kept in the socket send buffer along with data not yet transmitted. The snd_nxt field indicates the next sequence number to be sent and is incremented as data are transmitted. The area from snd_nxt to snd_una + snd_wnd is the remaining usable portion of the window, and its size determines whether additional data may be sent. The snd_nxt and snd_max values are normally maintained together except when TCP is retransmitting. The area between rcv_nxt and rcv_nxt + rcv_wnd is known as the receive window.

These variables are used in the output module to decide whether data can be sent, and in the input module to decide whether data that are received can be accepted. When the receiver detects that a packet is not acceptable because the data are all outside the window, it drops the packet but sends a copy of its most recent acknowledgment. If the packet contained old data, the first acknowledgment may have been lost, and thus it must be repeated. The acknowledgment also includes a window update, synchronizing the sender's state with the receiver's state.

If the TCP timestamp option is in use for the connection, the tests to see whether an incoming packet is acceptable are augmented with checks on the time-stamp. Each time that an incoming packet is accepted as the next expected packet, its timestamp is recorded in the ts_recent field in the TCP protocol control block. If an incoming packet includes a timestamp, the timestamp is compared to the most recently received timestamp. If the timestamp is less than the previous value, the packet is discarded as being an old duplicate and a current acknowledgment is sent in response. In this way, the timestamp serves as an extension to the sequence number, avoiding accidental acceptance of an old duplicate when the window is large or sequence numbers can be reused quickly. However, because of the granularity of the timestamp value, a timestamp received more than 24 days ago cannot be compared to a new value, and this test is bypassed. The current time is recorded when ts_recent is updated from an incoming timestamp to make this test. Of course, connections are seldom idle for longer than 24 days.


13.5. TCP Algorithms

Now that we have introduced TCP, its state machine, and its sequence space, we can begin to examine the implementation of the protocol in FreeBSD. Several aspects of the protocol implementation depend on the overall state of a connection. The TCP connection state, output state, and state changes depend on external events and timers. TCP processing occurs in response to one of three events:

  1. A request from the user, such as sending data, removing data from the socket receive buffer, or opening or closing a connection

  2. The receipt of a packet for the connection

  3. The expiration of a timer

These events are handled in the routines tcp_usr_send(), tcp_input(), and a set of timer routines. Each routine processes the current event and makes any required changes in the connection state. Then, for any transition that may require sending a packet, the tcp_output() routine is called to do any output that is necessary.

The criteria for sending a packet with data or control information are complicated, and therefore the TCP send policy is the most interesting and important part of the protocol implementation. For example, depending on the state- and flow-control parameters for a connection, any of the following may allow data to be sent that could not be sent previously:

  • A user send call that places new data in the send queue

  • The receipt of a window update from the peer

  • The expiration of the retransmission timer

  • The expiration of the window-update (persist) timer

In addition, the tcp_output() routine may decide to send a packet with control information, even if no data may be sent, for any of these reasons:

  • A change in connection state (e.g., open request, close request)

  • Receipt of data that must be acknowledged

  • A change in the receive window because of removal of data from the receive queue

  • A send request with urgent data

  • A connection abort

We shall consider most of these decisions in greater detail after we have described the states and timers involved. We begin with algorithms used for timing, connection setup, and shutdown; they are distributed through several parts of the code. We continue with the processing of new input and an overview of output processing and algorithms.

Timers

Unlike a UDP socket, a TCP connection maintains a significant amount of state information, and, because of that state, some operations must be done asynchronously. For example, data might not be sent immediately when a process presents them because of flow control. The requirement for reliable delivery implies that data must be retained after they are first transmitted so that they can be retransmitted if necessary. To prevent the protocol from hanging if packets are lost, each connection maintains a set of timers used to recover from losses or failures of the peer. These timers are stored in the protocol control block for a connection. The kernel provides a timer service via a set of callout() routines. The TCP module can register up to five timeout routines with the callout service, as shown in Table 13.3. Each routine has its own associated time at which it will be called. In earlier versions of BSD, timeouts were handled by the tcp_slowtimo() routine that was called every 500 milliseconds and would then do timer processing when necessary. Using the kernel's timer service is both more accurate, since each timer can be handled independently, and has less overhead, because no routine is called unless absolutely necessary.

Table 13.3. TCP timer routines.

Routine

Timeout

Description

tcp_timer_2msl

60s

wait on close

tcp_timer_keep

75s

send keep alive or drop dormant connection

tcp_timer_persist

5-60s

force a connection to persist

tcp_timer_rexmt

3ticks-64s

called when retransmission is necessary

tcp_timer_delack

100ms

send a delayed acknowledgement to the peer

Two timers are used for output processing. Whenever data are sent on a connection, the retransmit timer (tcp_rexmt()) is started by a call to callout_reset(), unless it is already running. When all outstanding data are acknowledged, the timer is stopped. If the timer expires, the oldest unacknowledged data are resent (at most one full-sized packet), and the timer is restarted with a longer value. The rate at which the timer value is increased (the timer backoff) is determined by a table of multipliers that provides an exponential increase in timeout values up to a ceiling.

The other timer used for maintaining output flow is the persist timer (tcp_timer_persist()) This timer protects against the other type of packet loss that could cause a connection to constipate: the loss of a window update that would allow more data to be sent. Whenever data are ready to be sent but the send window is too small to bother sending (zero, or less than a reasonable amount), and no data are already outstanding (the retransmit timer is not set), the persist timer is started. If no window update is received before the timer expires, the routine sends as large a segment as the window allows. If that size is zero, it sends a window probe (a single octet of data) and restarts the persist timer. If a window update was lost in the network, or if the receiver neglected to send a window update, the acknowledgment will contain current window information. On the other hand, if the receiver is still unable to accept additional data, it should send an acknowledgment for previous data with a still-closed window. The closed window might persist indefinitely; for example, the receiver might be a network-login client, and the user might stop terminal output and leave for lunch (or vacation).

The third timer used by TCP is a keepalive timer (tcp_timer_keep()) The keepalive timer has two different purposes at different phases of a connection. During connection establishment, this timer limits the time for the three-way handshake to complete. If the timer expires during connection setup, then the connection is closed. Once the connection completes, the keepalive timer monitors idle connections that might no longer exist on the peer because of a network partition or a crash. If a socket-level option is set and the connection has been idle since the most recent keepalive timeout, the timer routine will send a keepalive packet designed to produce either an acknowledgment or a reset (RST) from the peer TCP. If a reset is received, the connection will be closed; if no response is received after several attempts, the connection will be dropped. This facility is designed so that network servers can avoid languishing forever if the client disappears without closing. Keepalive packets are not an explicit feature of the TCP protocol. The packets used for this purpose by FreeBSD set the sequence number to 1 less than snd_una, which should elicit an acknowledgment from the peer if the connection still exists.

The fourth TCP timer is known as the 2MSL timer ("twice the maximum segment lifetime"). TCP starts this timer when a connection is completed by sending an acknowledgment for a FIN (from FIN_WAIT_2) or by receiving an ACK for a FIN (from CLOSING state, where the send side is already closed). Under these circumstances, the sender does not know whether the acknowledgment was received. If the FIN is retransmitted, it is desirable that enough state remain that the acknowledgment can be repeated. Therefore, when a TCP connection enters the TIME_WAIT state, the 2MSL timer is started; when the timer expires, the control block is deleted. If a retransmitted FIN is received, another ACK is sent, and the timer is restarted. To prevent this delay from blocking a process closing the connection, any process close request is returned successfully without the process waiting for the timer. Thus, a protocol control block may continue its existence even after the socket descriptor has been closed. In addition, FreeBSD starts the 2MSL timer when FIN_WAIT_2 state is entered after the user has closed; if the connection is idle until the timer expires, it will be closed. Because the user has already closed, new data cannot be accepted on such a connection in any case. This timer is set because certain other TCP implementations (incorrectly) fail to send a FIN on a receive-only connection. Connections to such hosts would remain in FIN_WAIT_2 state forever if the system did not have a timeout.

The final timer is the tcp_timer_delack(), which processes delayed acknowledgments. This will be described in Section 13.6.

Estimation of Round-Trip Time

When connections must traverse slow networks that lose packets, an important decision determining connection throughput is the value to be used when the retransmission timer is set. If this value is too large, data flow will stop on the connection for an unnecessarily long time before the dropped packet is resent. Another round-trip time interval is required for the sender to receive an acknowledgment of the resent segment and a window update, allowing it to send new data. (With luck, only one segment will have been lost, and the acknowledgment will include the other segments that had been sent.) If the timeout value is too small, however, packets will be retransmitted needlessly. If the cause of the network slowness or packet loss is congestion, then unnecessary retransmission only exacerbates the problem. The traditional solution to this problem in TCP is for the sender to estimate the round-trip time (rtt) for the connection path by measuring the time required to receive acknowledgments for individual segments. The system maintains an estimate of the round-trip time as a smoothed moving average, srtt [Postel, 1981b], using

In addition to a smoothed estimate of the round-trip time, TCP keeps a smoothed variance (estimated as mean difference, to avoid square-root calculations in the kernel). It employs an a value of 0.875 for the round-trip time and a corresponding smoothing factor of 0.75 for the variance. These values were chosen in part so that the system could compute the smoothed averages using shift operations on fixed-point values instead of floating-point values because on many hardware architectures it is expensive to use floating-point arithmetic. The initial retransmission timeout is then set to the current smoothed round-trip time plus four times the smoothed variance. This algorithm is substantially more efficient on long-delay paths with little variance in delay, such as satellite links, because it computes the BETA factor dynamically [Jacobson, 1988].

For simplicity, the variables in the TCP protocol control block allow measurement of the round-trip time for only one sequence value at a time. This restriction prevents accurate time estimation when the window is large; only one packet per window can be timed. However, if the TCP timestamps option is supported by both peers, a timestamp is sent with each data packet and is returned with each acknowledgment. Here, estimates of round-trip time can be obtained with each new acknowledgment; the quality of the smoothed average and variance is thus improved, and the system can respond more quickly to changes in network conditions.

Connection Establishment

There are two ways in which a new TCP connection can be established. An active connection is initiated by a connect call, whereas a passive connection is created when a listening socket receives a connection request. We consider each in turn.

The initial steps of an active connection attempt are similar to the actions taken during the creation of a UDP socket. The process creates a new socket, resulting in a call to the tcp_attach() routine. TCP creates an inpcb protocol control block and then creates an additional control block (a tcpcb structure), as described in Section 13.1. Some of the flow-control parameters in the tcpcb are initialized at this time. If the process explicitly binds an address or port number to the connection, the actions are identical to those for a UDP socket. Then a tcp_connect() call initiates the actual connection. The first step is to set up the association with in_pcbconnect(), again identically to this step in UDP. A packet-header template is created for use in construction of each output packet. An initial sequence number is chosen from a sequence-number prototype, which is then advanced by a substantial amount. The socket is then marked with soisconnecting(), the TCP connection state is set to TCPS_SYN_SENT, the keepalive timer is set (to 75 seconds) to limit the duration of the connection attempt, and tcp_output() is called for the first time.

The output-processing module tcp_output() uses an array of packet control flags indexed by the connection state to determine which control flags should be sent in each state. In the TCPS_SYN_SENT state, the SYN flag is sent. Because it has a control flag to send, the system sends a packet immediately using the prototype just constructed and including the current flow-control parameters. The packet normally contains three option fields: a maximum-segment-size option, a window-scale option, and a timestamps option (see Section 13.4). The maximum-segment-size option communicates the largest segment size that TCP is willing to accept. To compute this value, the system locates a route to the destination. If the route specifies a maximum transmission unit (MTU), the system uses that value after allowing for packet headers. If the connection is to a destination on a local network the maximum transmission unit of the outgoing network interface is used, possibly rounding down to a multiple of the mbuf cluster size for efficiency of buffering. If the destination is not local and nothing is known about the intervening path, the default segment size (512 octets) is used.

In earlier versions of FreeBSD many of the important variables relating to TCP connections, such as the MTU of the path between the two endpoints, and the data used to manage the connection were contained in the route entry that described the connection and in a routing entry as a set of route metrics. The TCP host cache was developed to centralize all this information in one easy-to-find place so that information that was gathered on one connection could be reused when a new connection was opened to the same endpoint. The data that is recorded on a connection is shown in Table 13.4. All the variables stored in a host cache entry are described in various parts of later sections of this chapter when they become relevant to our discussion of how TCP manages a connection.

Table 13.4. TCP host cache metrics.

Variable

Description

rmx_mtu

MTU for this path

rmx_ssthresh

outbound gateway buffer limit

rmx_rtt

estimated round-trip time

rmx_rttvar

estimated rtt variance

rmx_bandwidth

estimated bandwidth

rmx_cwnd

congestion window

rmx_sendpipe

outbound delay-bandwidth product

rmx_recvpipe

inbound delay-bandwidth product

Whenever a new connection is opened, a call is made to tcp_hc_get() to find any information on past connections. If an entry exists in the cache for the target endpoint, TCP uses the cached information to make better-informed decisions about managing the connection. When a connection is closed, the host cache is updated with all the relevant information that was discovered during the connection between the two hosts. Each host cache entry has a default lifetime of one hour. Anytime that the entry is accessed or updated, its lifetime is reset to one hour. Every five minutes the tcp_hc_purge() routine is called to clean out any entries that have passed their expiration time. Cleaning out old entries ensures that the host cache does not grow too large and that it always has reasonably fresh data.

TCP can use Path MTU Discovery as described in Mogul & Deering [1990]. Path MTU discovery is a process whereby the system probes the network to see what the maximum transfer unit is on a particular route between two nodes. It does this by sending packets with the IP flag don't fragment set on each packet. If the packet encounters a segment on its path to its destination on which it would have to be fragmented, then it is dropped by the intervening router, and an error is returned to the sender. The error message contains the maximum size packet that the segment will accept. This information is recorded in the TCP host cache for the appropriate endpoint and transmission is attempted with the smaller MTU. Once the connection is complete, because enough packets have made it through the network to establish a TCP connection, the revised MTU recorded in the host cache is confirmed. Packets will continue to be transmitted with the don't fragment flag set so that if the path to the node changes, and that path has an even smaller MTU, this new smaller MTU will be recorded. FreeBSD currently has no way of upgrading the MTU to a larger size when a route changes.

When a connection is first being opened, the retransmit timer is set to the default value (6 seconds) because no round-trip time information is available yet. With a bit of luck, a responding packet will be received from the target of the connection before the retransmit timer expires. If not, the packet is retransmitted and the retransmit timer is restarted with a greater value. If no response is received before the keepalive timer expires, the connection attempt is aborted with a "Connection timed out" error. If a response is received, however, it is checked for agreement with the outgoing request. It should acknowledge the SYN that was sent and should include a SYN. If it does both, the receive sequence variables are initialized, and the connection state is advanced to TCPS_ESTABLISHED. If a maximum-segment-size option is present in the response, the maximum segment size for the connection is set to the minimum of the offered size and the maximum transmission unit of the outgoing interface; if the option is not present, the default size (512 data bytes) is recorded. The flag TF_ACKNOW is set in the TCP control block before the output routine is called so that the SYN will be acknowledged immediately. The connection is now ready to transfer data.

The events that occur when a connection is created by a passive open are different. A socket is created and its address is bound as before. The socket is then marked by the listen call as willing to accept connections. When a packet arrives for a TCP socket in TOPS LISTEN state, a new socket is created with sonewconn(), which calls the TCP tcp_usr_attach() routine to create the protocol control blocks for the new socket. The new socket is placed on the queue of partial connections headed by the listening socket. If the packet contains a SYN and is otherwise acceptable, the association of the new socket is bound, both the send and the receive sequence numbers are initialized, and the connection state is advanced to TCPS_SYN_RECEIVED. The keepalive timer is set as before, and the output routine is called after TF_ACKNOW has been set to force the SYN to be acknowledged; an outgoing SYN is sent as well. If this SYN is acknowledged properly, the new socket is moved from the queue of partial connections to the queue of completed connections. If the owner of the listening socket is sleeping in an accept call or does a select, the socket will indicate that a new connection is available. Again, the socket is finally ready to send data. Up to one window of data may have already been received and acknowledged by the time that the accept call completes.

SYN Cache

One problem in previous implementations of TCP was that it was possible for a malicious program to flood a system with SYN packets, thereby preventing it from doing any useful work or servicing any real connections. This type of denial of service attack became common during the commercialization of the Internet in the late 1990s. To combat this attack, a syncache was introduced to efficiently store, and possibly discard, SYN packets that do not lead to real connections. The syncache handles the three-way handshake between a local server and connecting peers.

When a SYN packet is received for a socket that is in the LISTEN state, the TCP module attempts to add a new syncache entry for the packet using the syncache_add() routine. If there are any data in the received packet, they are not acknowledged at this time. Acknowledging the data would use up system resources, and an attacker could exhaust these resources by flooding the system with SYN packets that included data. If this SYN has not been seen before, a new entry is created in the hash table based on the packet's foreign address, foreign port, the local port of the socket, and a mask. The syncache module responds to the SYN with a SYN/ACK and sets a timer on the new entry. If the syncache contains an entry that matches the received packet, then it is assumed that the original SYN/ACK was not received by the peer initiating the connection, and another SYN/ACK is sent and the timer on the syncache entry is reset. There is no limit set on the number of SYN packets that can be sent by a connecting peer. Any limit would not follow the TCP RFCs and might impede connections over lossy networks.

Connection Shutdown

A TCP connection is symmetrical and full-duplex, so either side may initiate disconnection independently. As long as one direction of the connection can carry data, the connection remains open. A socket may indicate that it has completed sending data with the shutdown system call, which results in a call to the tcp_usr_shutdown() routine. The response to this request is that the state of the connection is advanced; from the ESTABLISHED state, the state becomes FIN_WAIT_1. The ensuing output call will send a FIN, indicating an end-of-file. The receiving socket will advance to CLOSE_WAIT but may continue to send. The procedure may be different if the process simply closes the socket. In that case, a FIN is sent immediately, but if new data are received, they cannot be delivered. Normally, higher-level protocols conclude their own transactions such that both sides know when to close. If they do not, however, TCP must refuse new data. It does so by sending a packet with the RST flag set if new data are received after the user has closed. If data remain in the send buffer of the socket when the close is done, TCP will normally attempt to deliver them. If the socket option SO_LINGER was set with a linger time of zero, the send buffer is simply flushed; otherwise, the user process is allowed to continue, and the protocol waits for delivery to conclude. Under these circumstances, the socket is marked with the state bit SS_NOFDREF (no file-descriptor reference). The completion of data transfer and the final close can take place an arbitrary amount of time later. When TCP finally completes the connection (or gives up because of timeout or other failure), it calls tcp_close(). The protocol control blocks and other dynamically allocated structures are freed at this time. The socket also is freed if the SS_NOFDREF flag has been set. Thus, the socket remains in existence as long as either a file descriptor or a protocol control block refers to it.


13.6. TCP Input Processing

Although TCP input processing is considerably more complicated than UDP input handling, the preceding sections have provided the background that we need to examine the actual operation. As always, the input routine is called with parameters

void tcp_input(
    struct mbuf *msg,
    int off0);

The first few steps probably are beginning to sound familiar:

1. Locate the TCP header in the received IP datagram. Make sure that the packet is at least as long as a TCP header, and use m_pullup() if necessary to make it contiguous.

2. Compute the packet length, set up the IP pseudo-header, and checksum the TCP header and data. Discard the packet if the checksum is bad.

3. Check the TCP header length; if it is larger than a minimal header, make sure that the whole header is contiguous.

4. Locate the protocol control block for the connection with the port number specified. If none exists, send a packet containing the reset flag RST and drop the packet.

5. Check whether the socket is listening for connections; if it is, follow the procedure described for passive connection establishment.

6. Process any TCP options from the packet header.

7. Clear the idle time for the connection, and set the keepalive timer to its normal value.

At this point, the normal checks have been made, and we are prepared to deal with data and control flags in the received packet. There are still many consistency checks that must be made during normal processing; for example, the SYN flag must be present if we are still establishing a connection and must not be present if the connection has been established. We shall omit most of these checks from our discussion, but the tests are important to prevent wayward packets from causing confusion and possible data corruption.

The next step in checking a TCP packet is to see whether the packet is acceptable according to the receive window. It is important that this step be done before control flags—in particular RST—are examined because old or extraneous packets should not affect the current connection unless they are clearly relevant in the current context. A segment is acceptable if the receive window has nonzero size and if at least some of the sequence space occupied by the packet falls within the receive window. If the packet contains data, some of the data must fall within the window. Portions of the data that precede the window are trimmed, since they have already been received, and portions that exceed the window also are discarded, since they have been sent prematurely. If the receive window is closed (rcv_wrul is zero), then only segments with no data and with a sequence number equal to rcv_nxt are acceptable. If an incoming segment is not acceptable, it is dropped after an acknowledgment is sent.

The processing of incoming TCP packets must be fully general, taking into account all the possible incoming packets and possible states of receiving end-points. However, the bulk of the packets processed falls into two general categories. Typical packets contain either the next expected data segment for an existing connection or an acknowledgment plus a window update for one or more data segments, with no additional flags or state indications. Rather than considering each incoming segment based on first principles, tcp_input() checks first for these common cases. This algorithm is known as header prediction. If the incoming segment matches a connection in the ESTABLISHED state, if it contains the ACK flag but no other flags, if the sequence number is the next value expected (and the timestamp, if any, is nondecreasing), if the window field is the same as in the previous segment, and if the connection is not in a retransmission state, then the incoming segment is one of the two common types. The system processes any timestamp option that the segment contains, recording the value received to be included in the next acknowledgment. If the segment contains no data, it is a pure acknowledgment with a window update. In the usual case, round-trip-timing information is sampled if it is available, acknowledged data are dropped from the socket send buffer, and the sequence values are updated. The packet is discarded once the header values have been checked. The retransmit timer is canceled if all pending data have been acknowledged; otherwise, it is restarted. The socket layer is notified if any process might be waiting to do output. Finally, tcp_output() is called because the window has moved forward, and that operation completes the handling of a pure acknowledgment.

If a packet meeting the tests for header prediction contains the next expected data, if no out-of-order data are queued for the connection, and if the socket receive buffer has space for the incoming data, then this packet is a pure insequence data segment. The sequencing variables are updated, the packet headers are removed from the packet, and the remaining data are appended to the socket receive buffer. The socket layer is notified so that it can notify any interested thread, and the control block is marked with a flag, indicating that an acknowledgment is needed. No additional processing is required for a pure data packet.

For packets that are not handled by the header-prediction algorithm, the processing steps are as follows:

1. Process the timestamp option if it is present, rejecting any packets for which the timestamp has decreased, first sending a current acknowledgment.

2. Check whether the packet begins before rcv_nxt. If it does, ignore any SYN in the packet, and trim any data that fall before rcv_nxt. If no data remain, send a current acknowledgment and drop the packet. (The packet is presumed to be a duplicate transmission.)

3. If the packet still contains data after trimming, and the process that created the socket has already closed the socket, send a reset (RST) and drop the connection. This reset is necessary to abort connections that cannot complete; it typically is sent when a remote-login client disconnects while data are being received.

4. If the end of the segment falls after the window, trim any data beyond the window. If the window was closed and the packet sequence number is rcv_nxt, the packet is treated as a window probe; TF_ACKNOW is set to send a current acknowledgment and window update, and the remainder of the packet is processed. If SYN is set and the connection was in TIME_WAIT state, this packet is really a new connection request, and the old connection is dropped; this procedure is called rapid connection reuse. Otherwise, if no data remain, send an acknowledgment and drop the packet.

The remaining steps of TCP input processing check the following flags and fields and take the appropriate actions: RST, ACK, window, URG, data, and FIN. Because the packet has already been confirmed to be acceptable, these actions can be done in a straightforward way:

5. If a timestamp option is present, and the packet includes the next sequence number expected, record the value received to be included in the next acknowledgment.

6. If RST is set, close the connection and drop the packet.

7. If ACK is not set, drop the packet.

8. If the acknowledgment-field value is higher than that of previous acknowledgments, new data have been acknowledged. If the connection was in SYN_RECEIVED state and the packet acknowledges the SYN sent for this connection, enter ESTABLISHED state. If the packet includes a timestamp option, use it to compute a round-trip time sample; otherwise, if the sequence range that was newly acknowledged includes the sequence number for which the round-trip time was being measured, this packet provides a sample. Average the time sample into the smoothed round-trip time estimate for the connection. If all outstanding data have been acknowledged, stop the retransmission timer; otherwise, set it back to the current timeout value. Finally, drop from the send queue in the socket the data that were acknowledged. If a FIN has been sent and was acknowledged, advance the state machine.

9. Check the window field to see whether it advances the known send window. First, check whether this packet is a new window update. If the sequence number of the packet is greater than that of the previous window update, or the sequence number is the same but the acknowledgment-field value is higher, or if both sequence and acknowledgment are the same but the window is larger, record the new window.

10. If the urgent-data flag URG is set, compare the urgent pointer in the packet to the last-received urgent pointer. If it is different, new urgent data have been sent. Use the urgent pointer to compute so_oobmark, the offset from the beginning of the socket receive buffer to the urgent mark (Section 11.6), and notify the socket with sohasoutofband(). If the urgent pointer is less than the packet length, the urgent data have all been received. TCP normally removes the final data octet sent in urgent mode (the last octet before the urgent pointer) and places that octet in the protocol control block until it is requested with a PRU_RCVOOB request. (The end of the urgent data is a subject of disagreement; the BSD interpretation follows the original TCP specification.) A socket option, SO_OOBINLINE, may request that urgent data be left in the queue with the normal data, although the mark on the data stream is still maintained.

11. At long last, examine the data field in the received packet. If the data begin with rcv_nxt, then they can be placed directly into the socket receive buffer with sbappendstream(). The flag TF_DELACK is set in the protocol control block to indicate that an acknowledgment is needed, but the latter is not sent immediately in hope that it can be piggybacked on any packets sent soon (presumably in response to the incoming data) or combined with acknowledgment of other data received soon; see the subsection on delayed acknowledgments and window updates in Section 13.7. If no activity causes a packet to be returned before the next time that the tcp_delack() routine runs, it will change the flag to TF_ACKNOW and call the tcp_output() routine to send the acknowledgment. Acknowledgments can thus be delayed by no more than 200 milliseconds. If the data do not begin with rcv_nxt, the packet is retained in a per-connection queue until the intervening data arrive, and an acknowledgment is sent immediately.

12. As the final step in processing a received packet, check for the FIN flag. If it is present, the connection state machine may have to be advanced, and the socket is marked with socantrcvmore() to convey the end-of-file indication. If the send side has already closed (a FIN was sent and acknowledged), the socket is now considered closed, and it is so marked with soisdisconnected(). The TF_ACKNOW flag is set to force immediate acknowledgment.

Step 10 completes the actions taken when a new packet is received by tcp_input(). However, as noted earlier in this section, receipt of input may require new output. In particular, acknowledgment of all outstanding data or a new window update requires either new output or a state change by the output module. Also, several special conditions set the TF_ACKNOW flag. In these cases, tcp_output() is called at the conclusion of input processing.


13.7. TCP Output Processing

We are finally ready to investigate the most interesting part of the TCP implementation: the send policy. As we saw earlier, a TCP packet contains an acknowledgment and a window field as well as data, and a single packet may be sent if any of these three fields change. A naive TCP send policy might send many more packets than necessary. For example, consider what happens when a user types one character to a remote-terminal connection that uses remote echo. The server-side TCP receives a single-character packet. It might send an immediate acknowledgment of the character. Then, milliseconds later, the login server would read the character, removing the character from the receive buffer; the TCP might immediately send a window update, noting that one additional octet of send window was available. After another millisecond or so, the login server would send an echoed character back to the client, necessitating a third packet sent in response to the single character of input. It is obvious that all three responses (the acknowledgment, the window update, and the data return) could be sent in a single packet. However, if the server were not echoing input data, the acknowledgment could not be withheld for too long a time or the client-side TCP would begin to retransmit. The algorithms used in the send policy to minimize network traffic yet to maximize throughput are the most subtle part of a TCP implementation. The send policy used in FreeBSD includes several standard algorithms, as well as a few approaches suggested by the network research community. We shall examine each part of the send policy.

As we saw in the previous section, there are several different events that may trigger the sending of data on a connection; in addition, packets must be sent to communicate acknowledgments and window updates (consider a one-way connection!).

Sending of Data

The most obvious reason that the tcp output module tcp_output() is called is that the user has written new data to the socket. Write operations are done with a call to tcp_usr_send() routine. (Recall that sosend() waits for enough space in the socket send buffer if necessary and then copies the user's data into a chain of mbufs that is passed to the protocol by the tcp_usr_send() routine.) The action in tcp_usr_send() is simply to place the new output data in the socket's send buffer with sbappendstream() and to call tcp_output(). If flow control permits, tcp_output() will send the data immediately.

The actual send operation is not substantially different from one for a UDP datagram socket. The differences are that the header is more complicated and additional fields must be initialized, and the data sent are simply a copy of the user's data. However, for send operations large enough for sosend() to place the data in external mbuf clusters, the copy is done by creating a new reference to the data cluster. A copy must be retained in the socket's send buffer in case retransmission is required. Also, if the number of data octets is larger than the size of a single maximum-sized segment, multiple packets will be constructed and sent in a single call.

The tcp_output() routine allocates an mbuf to contain the output packet header and copies the contents of the header template into that mbuf. If the data to be sent fit into the same mbuf as the header, tcp_output() copies them into place from the socket send buffer using the m_copydata() routine. Otherwise, tcp_output() adds the data to be sent as a separate chain of mbufs obtained with an m_copy() operation from the appropriate part of the send buffer. The sequence number for the packet is set from snd_nxt, and the acknowledgment is set from rcv_nxt. The flags are obtained from an array containing the flags to be sent in each connection state. The window to be advertised is computed from the amount of space remaining in the socket's receive buffer; however, if that amount is small (less than one-fourth of the buffer and less than one segment), it is set to zero. The window is never allowed to end at a smaller sequence number than the one in which it ended in the previous packet. If urgent data have been sent, the urgent pointer and flag are set accordingly. One other flag must be set: The PUSH flag on a packet indicates that data should be passed to the user; it is like a buffer-flush request. This flag is generally considered obsolete but is set whenever all the data in the send buffer have been sent; FreeBSD ignores this flag on input. Once the header is filled in, the packet is checksummed. The remaining parts of the IP header are initialized, including the type-of-service and time-to-live fields, and the packet is sent with ip_output(). The retransmission timer is started if it is not already running, and the snd_nxt and snd_max values for the connection are updated.

Avoidance of the Silly-Window Syndrome

Silly-window syndrome is the name given to a potential problem in a window-based flow-control scheme in which a system sends several small packets rather than waiting for a reasonable-sized window to become available [Clark, 1982]. For example, if a network-login client program has a total receive buffer size of 4096 octets, and the user stops terminal output during a large printout, the buffer will become nearly full as new full-sized segments are received. If the remaining buffer space dropped to 10 bytes, it would not be useful for the receiver to volunteer to receive an additional 10 octets. If the user then allowed a few characters to print and stopped output again, it still would not be useful for the receiving TCP to send a window update allowing another 14 octets. Instead, it is desirable to wait until a reasonably large packet can be sent, since the receive buffer already contains enough data for the next several pages of output. Avoidance of the silly-window syndrome is desirable in both the receiver and the sender of a flow-controlled connection, as either end can prevent silly small windows from being used. We described receiver avoidance of the silly-window syndrome in the previous subsection; when a packet is sent, the receive window is advertised as zero if it is less than one packet and less than one-fourth of the receive buffer. For sender avoidance of the silly-window syndrome, an output operation is delayed if at least a full packet of data is ready to be sent but less than one full packet can be sent because of the size of the send window. Instead of sending, tcp_output() sets the output state to persist state by starting the persist timer. If no window update has been received by the time that the timer expires, the allowable data are sent in the hope that the acknowledgment will include a larger window. If it does not, the connection stays in persist state, sending a window probe periodically until the window is opened.

An initial implementation of sender avoidance of the silly-window syndrome produced large delays and low throughput over connections to hosts using TCP implementations with tiny buffers. Unfortunately, those implementations always advertised receive windows less than the maximum segment size—a behavior that was considered silly by this implementation. As a result of this problem, the TCP keeps a record of the largest receive window offered by a peer in the protocol-control-block variable max_sndwnd. When at least one-half of max_sndwnd may be sent, a new segment is sent. This technique improved performance when a BSD system was communicating with these limited hosts.

Avoidance of Small Packets

Network traffic exhibits a bimodal distribution of sizes. Bulk data transfers tend to use the largest possible packets for maximum throughput. Interactive services (such as network-login) tend to use small packets, however, often containing only a single data character. On a fast local-area network, such as an Ethernet, the use of single-character packets generally is not a problem because the network bandwidth usually is not saturated. On long-haul networks interconnected by slow or congested links, or wireless LANs that are both slow and lossy, it is desirable to collect input over some period and then send it in a single network packet. Various schemes have been devised for collecting input over a fixed time—usually about 50 to 100 milliseconds—and then sending it in a single packet. These schemes noticeably slow character echo times on fast networks, however, and often save few packets on slow networks. In contrast, a simple and elegant scheme for reducing small-packet traffic was suggested by Nagle [1984]. This scheme allows the first octet output to be sent alone in a packet with no delay. Until this packet is acknowledged, however, no new small packets may be sent. If enough new data arrive to fill a maximum-sized packet, another packet is sent. As soon as the outstanding data are acknowledged, the input that was queued while waiting for the first packet may be sent. Only one small packet may ever be outstanding on a connection at one time. The net result is that data from small output operations are queued during one round-trip time. If the round-trip time is less than the intercharacter arrival time, as it is in a remote-terminal session on a local-area network, transmissions are never delayed, and response time remains low. When a slow network intervenes, input after the first character is queued, and the next packet contains the input received during the preceding round-trip time. This algorithm is attractive because of both its simplicity and its self-tuning nature.

Eventually, people discovered that this algorithm did not work well for certain classes of network clients that sent streams of small requests that could not be batched. One such client was the network-based X Window System [Scheifler & Gettys, 1986], which required immediate delivery of small messages to get real-time feedback for user interfaces such as rubber-banding to sweep out a new window. Hence, the developers added an option, TCP_NODELAY, to defeat this algorithm on a connection. This option can be set with a setsockopt call, which reaches TCP via the tcp_ctloutput() routine. Unfortunately, the X Window System library always sets the TCP_NODELAY flag, rather than only when the client is using mouse-driven positioning.

Delayed Acknowledgments and Window Updates

TCP packets must be sent for reasons other than data transmission. On a one-way connection, the receiving TCP must still send packets to acknowledge received data and to advance the sender's send window. The mechanism for delaying acknowledgments in hope of piggybacking or coalescing them with data or window updates was described in Section 13.6. In a bulk data transfer, the time at which window updates are sent is a determining factor for network throughput. For example, if the receiver simply set the TF_DELACK flag each time that data were received on a bulk-data connection, acknowledgments would be sent every 200 milliseconds. If 8192-octet windows are used on a 10-Mbit/s Ethernet, this algorithm will result in a maximum throughput of 320 Kbit/s, or 3.2 percent of the physical network bandwidth. Clearly, once the sender has filled the send window that it has been given, it must stop until the receiver acknowledges the old data (allowing them to be removed from the send buffer and new data to replace them) and provides a window update (allowing the new data to be sent).

Because TCP's window-based flow control is limited by the space in the socket receive buffer, TCP has the PR_RCVD flag set in its protocol-switch entry so that the protocol will be called (via the tcp_usr_rcvd() routine) when the user has done a receive call that has removed data from the receive buffer. The tcp_usr_rcvd() routine simply calls tcp_output(). Whenever tcp_output() determines that a window update sent under the current circumstances would provide a new send window to the sender large enough to be worthwhile, it sends an acknowledgment and window update. If the receiver waited until the window was full, the sender would already have been idle for some time when it finally received a window update. Furthermore, if the send buffer on the sending system was smaller than the receiver's buffer—and thus smaller than the receiver's window—the sender would be unable to fill the receiver's window without receiving an acknowledgment. Therefore, the window-update strategy in FreeBSD is based on only the maximum segment size. Whenever a new window update would move the window forward by at least two full-sized segments, the window update is sent. This window-update strategy produces a twofold reduction in acknowledgment traffic and a twofold reduction in input processing for the sender. However, updates are sent often enough to give the sender feedback on the progress of the connection and to allow the sender to continue sending additional segments.

Note that TCP is called at two different stages of processing on the receiving side of a bulk data transfer: It is called on packet reception to process input, and it is called after each receive operation removing data from the input buffer. At the first call, an acknowledgment could be sent, but no window update could be sent. After the receive operation, a window update also is possible. Thus, it is important that the algorithm for updates run in the second half of this cycle.

Retransmit State

When the retransmit timer expires while a sender is awaiting acknowledgment of transmitted data, tcp_output() is called to retransmit. The retransmit timer is first set to the next multiple of the round-trip time in the backoff series. The variable snd_nxt is moved back from its current sequence number to snd_una. A single packet is then sent containing the oldest data in the transmit queue. Unlike some other systems, FreeBSD does not keep copies of the packets that have been sent on a connection; it retains only the data. Thus, although only a single packet is retransmitted, that packet may contain more data than does the oldest outstanding packet. On a slow connection with small send operations, such as a remote login, this algorithm may cause a single-octet packet that is lost to be retransmitted with all the data queued since the initial octet was first transmitted.

If a single packet was lost in the network, the retransmitted packet will elicit an acknowledgment of all data transmitted thus far. If more than one packet was lost, the next acknowledgment will include the retransmitted packet and possibly some of the intervening data. It may also include a new window update. Thus, when an acknowledgment is received after a retransmit timeout, any old data that were not acknowledged will be resent as though they had not yet been sent, and some new data may be sent as well.

Slow Start

Many TCP connections traverse several networks between source and destination. When some of the networks are slower than others, the entry router to the slowest network often is presented with more traffic than it can handle. It may buffer some input packets to avoid dropping them because of sudden changes in flow, but eventually its buffers will fill and it must begin dropping packets. When a TCP connection first starts sending data across a fast network to a router forwarding via a slower network, it may find that the router's queues are already nearly full. In the original send policy used in BSD, a bulk-data transfer would start out by sending a full window of packets once the connection was established. These packets could be sent at the full speed of the network to the bottleneck router, but that router could transmit them only at a much slower rate. As a result, the initial burst of packets was highly likely to overflow the router's queue, and some of the packets would be lost. If such a connection used an expanded window size in an attempt to gain performance—for example, when traversing a satellite-based network with a long round-trip time—this problem would be even more severe. However, if the connection could once reach steady state, a full window of data often could be accommodated by the network if the packets were spread evenly throughout the path. At steady state, new packets would be injected into the network only when previous packets were acknowledged, and the number of packets in the network would be constant. Figure 13.9 shows the desired steady state. In addition, even if packets arrived at the outgoing router in a cluster, they would be spread out when the network was traversed by at least their transmission times in the slowest network. If the receiver sent acknowledgments when each packet was received, the acknowledgments would return to the sender with approximately the correct spacing. The sender would then have a self-clocking means for transmitting at the correct rate for the network without sending bursts of packets that the bottleneck could not buffer.

Figure 13.9. Acknowledgment clocking. There are two routers connected by a slow link between the sender and the receiver. The thickness of the links represents their speed. The width of the packets represents their time to travel down the link. Fast links are wide and the packets are narrow. Slow links are narrow and the packets are wide. In the steady state shown, the sender sends a new packet each time an acknowledgment is received from the receiver.

[View full size image]

An algorithm named slow start brings a TCP connection to this steady state [Jacobson, 1988]. It is called slow start because it is necessary to start data transmission slowly when traversing a slow network. Figure 13.10 shows the progress of the slow-start algorithm. The scheme is simple: A connection starts out with a limit of just one outstanding packet. Each time that an acknowledgment is received, the limit is increased by one packet. If the acknowledgment also carries a window update, two packets can be sent in response. This process continues until the window is fully open. During the slow-start phase of the connection, if each packet was acknowledged separately, the limit would be doubled during each exchange, resulting in an exponential opening of the window. Delayed acknowledgments might cause acknowledgments to be coalesced if more than one packet could arrive at the receiver within 200 milliseconds, slowing the window opening slightly. However, the sender never sends bursts of more than two or three packets during the opening phase and sends only one or two packets at a time once the window has opened.

Figure 13.10. The progression of the slow-start algorithm.

[View full size image]

The implementation of the slow-start algorithm uses a second window, like the send window but maintained separately, called the congestion window (snd_cwnd). The congestion window is maintained according to an estimate of the data that the network is currently able to buffer for this connection. The send policy is modified so that new data are sent only if allowed by both the normal and congestion send windows. The congestion window is initialized to the size of one packet, causing a connection to begin with a slow start. It is set to one packet whenever transmission stops because of a timeout. Otherwise, once a retransmitted packet was acknowledged, the resulting window update might allow a full window of data to be sent, which would once again overrun intervening routers. This slow start after a retransmission timeout eliminates the need for a test in the output routine to limit output to one packet on the initial timeout. In addition, the timeout may indicate that the network has become slower because of congestion, and temporary reduction of the window may help the network to recover from its condition. The connection is forced to reestablish its clock of acknowledgments after the connection has come to a halt, and the slow start has this effect as well. A slow start is also forced if a connection begins to transmit after an idle period of at least the current retransmission value (a function of the smoothed round-trip time and variance estimates).

Source-Quench Processing

If a router along the route used by a connection receives more packets than it can send along this path, it will eventually be forced to drop packets. When packets are dropped, the router may send an ICMP source quench error message to hosts whose packets have been dropped to indicate that the senders should slow their transmissions. Although this message indicates that some change should be made, it provides no information on how much of a change must be made or for how long the change should take effect. In addition, not all routers send source-quench messages for each packet dropped. The use of the slow-start algorithm after retransmission timeouts allows a connection to respond correctly to a dropped packet whether or not a source quench is received to indicate the loss. The action on receipt of a source quench for a TCP connection is simply to anticipate the timeout because of the dropped packet, setting the congestion window to one packet. This action prevents new packets from being sent until the dropped packet is resent at the next timeout. At that time, the slow start will begin again.

Buffer and Window Sizing

The performance of a TCP connection is obviously limited by the bandwidth of the path that the connection must transit. The performance is also affected by the round-trip time for the path. For example, paths that traverse satellite links have a long intrinsic delay, even though the bandwidth may be high, but the throughput is limited to one window of data per round-trip time. After filling the receiver's window, the sender must wait for at least one round-trip time for an acknowledgment and window update to arrive. To take advantage of the full bandwidth of a path, both the sender and receiver must use buffers at least as large as the bandwidth-delay product to allow the sender to transmit during the entire round-trip time. In steady state, this buffering allows the sender, receiver, and intervening parts of the network to keep the pipeline filled at each stage. For some paths, using slow start and a large window can lead to much better performance than could be achieved previously.

The round-trip time for a network path includes two components: transit time and queueing time. The transit time comprises the propagation, switching, and forwarding time in the physical layers of the network, including the time to transmit packets bit by bit after each store-and-forward hop. Ideally, queueing time would be negligible, with packets arriving at each node of the network just in time to be sent after the preceding packet. This ideal flow is possible when a single connection using a suitable window size is synchronized with the network. However, as additional traffic is injected into the network by other sources, queues build up in routers, especially at the entrance to the slower links in the path. Although queueing delay is part of the round-trip time observed by each network connection that is using a path, it is not useful to increase the operating window size for a connection to a value larger than the product of the limiting bandwidth for the path times the transit delay. Sending additional data beyond that limit causes the additional data to be queued, increasing queueing delay without increasing throughput.

Avoidance of Congestion with Slow Start

The addition of the slow-start algorithm to TCP allows a connection to send packets at a rate that the network can tolerate, reaching a steady state at which packets are sent only when another packet has exited the network. A single connection may reasonably use a large window without flooding the entry router to the slow network on startup. As a connection opens the window during a slow start, it injects packets into the network until the network links are kept busy. During this phase, it may send packets at up to twice the rate at which the network can deliver data because of the exponential opening of the window. If the window is chosen appropriately for the path, the connection will reach steady state without flooding the network. However, with multiple connections sharing a path, the bandwidth available to each connection is reduced. If each connection uses a window equal to the bandwidth-delay product, the additional packets in transit must be queued, increasing delay. If the total offered load is too high, routers must drop packets rather than increasing the queue sizes and delay. Thus, the appropriate window size for a TCP connection depends not only on the path, but also on competing traffic. A window size large enough to give good performance when a long-delay link is in the path will overrun the network when most of the round-trip time is in queueing delays. It is highly desirable for a TCP connection to be self-tuning, as the characteristics of the path are seldom known at the endpoints and may change with time. If a connection expands its window to a value too large for a path, or if additional load on the network collectively exceeds the capacity, router queues will build until packets must be dropped. At this point, the connection will close the congestion window to one packet and will initiate a slow start. If the window is simply too large for the path, however, this process will repeat each time that the window is opened too far.

The connection can learn from this problem and can adjust its behavior using another algorithm associated with the slow-start algorithm. This algorithm keeps a state variable for each connection, snd_ssthresh (slow-start threshold), which is an estimate of the usable window for the path. When a packet is dropped, as evidenced by a retransmission timeout, this window estimate is set to one-half the number of the outstanding data octets. The current window is obviously too large at the moment, and the decrease in window utilization must be large enough that congestion will decrease rather than stabilizing. At the same time, the slow-start window (snd_cwnd) is set to one segment to restart. The connection starts up as before, opening the window exponentially until it reaches the snd_ssthresh limit. At this point, the connection is near the estimated usable window for the path. It enters steady state, sending data packets as allowed by window updates. To test for improvement in the network, it continues to expand the window slowly; as long as this expansion succeeds, the connection can continue to take advantage of reduced network load. The expansion of the window in this phase is linear, with one additional full-sized segment being added to the current window for each full window of data transmitted. This slow increase allows the connection to discover when it is safe to resume use of a larger window while reducing the loss in throughput because of the wait after the loss of a packet before transmission can resume. Note that the increase in window size during this phase of the connection is linear as long as no packets are lost, but the decrease in window size when signs of congestion appear is exponential (it is divided by 2 on each timeout). With the use of this dynamic window-sizing algorithm, it is possible to use larger default window sizes for connection to all destinations without overrunning networks that cannot support them.

Fast Retransmission

Packets can be lost in the network for two reasons: congestion and corruption. In either case, TCP detects lost packets by a timeout, causing a retransmission. When a packet is lost, the flow of packets on a connection comes to a halt while waiting for the timeout. Depending on the round-trip time and variance, this timeout can result in a substantial period during which the connection makes no progress. Once the timeout occurs, a single packet is retransmitted as the first phase of a slow start, and the slow-start threshold is set to one-half the previous operating window. If later packets are not lost, the connection goes through a slow startup to the new threshold, and it then gradually opens the window to probe whether any congestion has disappeared. Each of these phases lowers the effective throughput for the connection. The result is decreased performance, even though the congestion may have been brief.

When a connection reaches steady state, it sends a continuous stream of data packets in response to a stream of acknowledgments with window updates. If a single packet is lost, the receiver sees packets arriving out of order. Most TCP receivers, including FreeBSD, respond to an out-of-order segment with a repeated acknowledgment for the in-order data. If one packet is lost while enough packets to fill the window are sent, each packet after the lost packet will provoke a duplicate acknowledgment with no data, window update, or other new information. The receiver can infer the out-of-order arrival of packets from these duplicate acknowledgments. Given enough evidence of reordering, the receiver can assume that a packet has been lost. The FreeBSD TCP implements fast retransmission based on this signal. Figure 13.11 shows the sequence of packet transmissions and acknowledgments when using the fast-retransmission algorithm during the loss of a single packet. After detecting three identical acknowledgments, the tcp_input () function saves the current connection parameters, simulates a retransmission timeout to resend one segment of the oldest data in the send queue, and then restores the current transmit state. Because this indication of a lost packet is a congestion signal, the estimate of the network buffering limit, snd_ssthresh, is set to one-half of the current window. However, because the stream of acknowledgments has not stopped, a slow start is not needed. If a single packet has been lost, doing fast retransmission fills in the gap more quickly than would waiting for the retransmission timeout. An acknowledgment for the missing segment, plus all out-of-order segments queued before the retransmission, will then be received, and the connection can continue normally.

Figure 13.11. Fast retransmission. The thick, longer boxes represent the data packets being sent. The thin, shorter lines represent the acknowledgments being returned.

Even with fast retransmission, it is likely that a TCP connection that suffers a lost segment will reach the end of the send window and be forced to stop transmission while awaiting an acknowledgment for the lost segment. However, after the fast retransmission, duplicate acknowledgments are received for each additional packet received by the peer after the lost packet. These duplicate acknowledgments imply that a packet has left the network and is now queued by the receiver. In that case, the packet does not need to be considered as within the network congestion window, possibly allowing additional data to be sent if the receiver's window is large enough. Each duplicate acknowledgment after a fast retransmission thus causes the congestion window to be moved forward artificially by the segment size. If the receiver's window is large enough, it allows the connection to make forward progress during a larger part of the time that the sender awaits an acknowledgment for the retransmitted segment. For this algorithm to have effect, the sender and receiver must have additional buffering beyond the normal bandwidth-delay product; twice that amount is needed for the algorithm to have full effect.


13.8. Internet Control Message Protocol (ICMP)

The Internet Control Message Protocol (ICMP) [Postel, 1981c] is the control- and error-message protocol for IPv4. Although it is layered above IPv4 for input and output operations, much like UDP, it is really an integral part of IPv4. Unlike those of UDP, most ICMP messages are received and implemented by the kernel. ICMP messages may also be sent and received via a raw IPv4 socket (see Section 12.7).

ICMP messages fall into three general classes. One class includes various errors that may occur somewhere in the network and that may be reported back to the originator of the packet provoking the error. Such errors include routing failures (network or host unreachable), expiration of the time-to-live field in a packet, or a report by the destination host that the target protocol or port number is not available. Error packets include the IPv4 header plus at least eight additional octets of the packet that encountered the error. The second message class may be considered as router-to-host control messages. Instances of such messages are the source-quench message that reports packet loss caused by excessive output, the routing redirect message that informs a host that a better route is available for a host or network via a different router, and a router advertisements that provides a simple way for a host to discover its router. The final message class includes network management, testing, and measurement packets. These packets include a network-address request and reply, a network-mask request and reply, an echo request and reply, a timestamp request and reply, and a generic information request and reply.

All the actions and replies required by an incoming ICMP message are done by the ICMP module. ICMP packets are received from IPv4 via the normal protocol-input entry point because ICMP has its own IPv4 protocol number. The ICMP input routine handles three major cases. If the packet is an error, such as port unreachable, then the message is processed and delivered to any higher-level protocol that might need to know it, such as the one that initiated the communication. Messages that require a response—for example, an echo—are processed and then sent back to their source with the icmp_reflect() routine. Finally, if there are any sockets listening for ICMP messages, they are given a copy of the message by a call to rip_input() at the end of the icmp_input() routine.

When error indications or source quenches are received, a generic address is constructed in a sockaddr structure. The address and error code are reported to each network protocol's control-input entry, pr_ctlinput(), by the icmp_input() routine. For example, an ICMP port unreachable message causes errors for only those connections with the indicated remote port and protocol.

Routing changes indicated by redirect messages are processed by the rtredirect() routine. It verifies that the router from which the message was received was the next-hop gateway in use for the destination, and it checks that the new gateway is on a directly attached network. If these tests succeed, the kernel routing tables are modified accordingly. If the new route is of equivalent scope to the previous route (e.g., both are for the destination network), the gateway in the route is changed to the new gateway. If the scope of the new route is smaller than that of the original route (either a host redirect is received when a network route was used, or the old route used a wildcard route), a new route is created in the kernel table. Routes that are created or modified by redirects are marked with the flags RTF_DYNAMIC and RTF_MODIFIED, respectively. Once the routing tables are updated, the protocols are notified by pfctlinput(), using a redirect code rather than an error code. TCP and UDP both ignore the redirect message because they do not store a pointer to the route. The next packet sent on the socket will reallocate a route, choosing the new route if that one is now the best route.

Once an incoming ICMP message has been processed by the kernel, it is passed to rip_input() for reception by any ICMP raw sockets. The raw sockets can also be used to send ICMP messages. The low-level network test program ping works by sending ICMP echo requests on a raw socket and listening for corresponding replies.

ICMP is also used by other Internet network protocols to generate error messages. UDP sends only ICMP port unreachable error messages, and TCP uses other means to report such errors. However, many different errors may be detected by IP, especially on systems used as IP routers. The icmp_error() function constructs an error message of a specified type in response to an IP packet. Most error messages include a portion of the original packet that caused the error, as well as the type and code for the error. The source address for the error packet is selected according to the context. If the original packet was sent to a local system address, that address is used as the source. Otherwise, an address is used that is associated with the interface on which the packet was received, as when forwarding is done; the source address of the error message can then be set to the address of the router on the network closest to (or shared with) the originating host. Also, when IP forwards a packet via the same network interface on which that packet was received, it may send a redirect message to the originating host if that host is on the same network. The icmp_error() routine accepts an additional parameter for redirect messages: the address of the new router to be used by the host.


13.9. 1Pv6

After many successful years of deploying and using IPv4, several issues arose that caused the Internet community to start working on new versions of the Internet protocols. The driving force behind this work was that the original Internet was running out of addresses [Gross & Almquist, 1992]. Several solutions had been proposed and implemented within the IPv4 protocols to handle this problem, including subnetting, and Classless Inter-Domain Routing (CIDR), [Fuller et al., 1993; Mogul & Postel, 1985] but neither of them proved sufficient. Several different proposals were made to completely replace the IPv4 protocols, and it took several years to make a final decision. Work on the new generation of the Internet protocols has been proceeding since the early 1990s, but it was not until 2003 that the protocol was rolled out by any large vendors. To date, the adoption of the new protocols has been limited because of the huge installed base of IPv4 that must be converted.

FreeBSD includes an IPv6 networking domain that contains an implementation of the IPv6 protocols. The domain supports the entire suite of protocols from the network through the transport layers. The protocols are described in a large set of RFCs starting with Deering & Hinden [1998a]. During the development of IPv6, several open-source implementations were written. Each implementation supported a different subset of the full features of IPv6 according to the needs of its authors. The one that eventually had the most complete set was developed by the KAME project [KAME, 2003] and is the implementation that was adopted by FreeBSD.

A complete discussion of IPv6 is beyond the scope of this book. This section discusses the areas of IPv6 that make it different from IPv4 and the changes that had to be made to FreeBSD to accommodate them.

There are several major differences between IPv4 and IPv6 including:

  • 128-bit addresses at the network layer

  • Emphasis on automatic configuration

  • Native support for security protocols

Since a factor driving the move to a new protocol was the need for more addresses, the first change to be made between IPv4 and IPv6 was to enlarge the size of an address. In IPv4, an address, which identifies a unique interface in the network, is 32 bits. This size is theoretically large enough to address over four billion interfaces. There are several reasons why that theoretical maximum is never reached. First is the need to control the size of the routing tables in the core Internet routers. Internet routing is most efficient when many addresses can be communicated by a single address, the address of the router to that network. If each address required its own route, there would be over four billion addresses in every routing table in the Internet, which would not be possible given the current state of network hardware and software. Thus, addresses are aggregated into blocks, and these blocks are assigned to ISPs, who then carve them up into smaller blocks for their customers. The customers then take these blocks and break them down further, through subnetting, and finally assign individual addresses to particular computers. At each level of this hierarchy, some addresses are kept aside for future use, which leads to the second source of IP address waste, overallocation. Because it is expensive and difficult to renumber a large installation of machines, customers request more addresses than they will ever need in an attempt to prevent ever having to renumber their networks. This overallocation has lead to several calls for companies and ISPs to return unused addresses [Nesser, 1996]. For these reasons, the size of an IP address was extended to 128 bits. The number of addresses available in IPv6 has been compared to numbering all the atoms in the universe or giving every person on the earth over a billion IP addresses.

As the Internet has been embraced by people who are not computer scientists and engineers, a major stumbling block has been the difficulty of setting up and maintaining even a single host in the Internet. Companies have teams of professionals who do this work, but for a small company—for example, a dentist's office or a sole proprietorship—the task can be daunting. These difficulties led the designers of IPv6 to include several types of autoconfiguration into the protocol. Ideally, anyone using IPv6 can turn on a computer, connect a network cable to it, and be on the network in a matter of minutes. This goal has not been achieved, but it does explain many of the design decisions in the IPv6 protocols.

Even before the Internet was a commercial success, network researchers and operators understood that the original protocols did not provide any form of security to users of the network. The lack of security was because the environment of the original Internet was one of cooperation, in which the emphasis was on sharing information. IPv6 includes a set of security protocols (IPSec) that are present in IPv4 as well. These protocols are a standard part of IPv6 and are covered in Section 13.10.

IPv6 Addresses

The 128-bit addresses in IPv6 necessitated creating new structures to hold them and new interfaces to handle them. While it is reasonably easy to work with the traditional dotted quad notation of IPv4 (i.e., 128.32.1.1), writing out an IPv6 address textually requires a bit more work, which is why the addressing architecture of IPv6 received its own RFC [Deering & Hinden, 1998b]. IPv6 defines several types of addresses:

Unicast | Just like a unicast address in IPv4, the IPv6 unicast address is a 128-bit quantity that uniquely identifies an interface on a host.
Multicast | An address that identifies a set of interfaces, usually participating in some form of group communication. A packet sent to this address is delivered to all interfaces in the network that are bound to this address.
Anycast | Anycast addresses are used to identify common services. The network will route a packet sent to this address to the nearest interface bound to that address. Nearest is measured by the number of hops the packet would have to make between the source and destination.

Note that unlike IPv4, IPv6 does not have the concept of a broadcast address that is received by all interfaces on a particular link. The role of broadcast addresses in IPv4 is to provide a way for hosts to discover services even when they do not yet have their own IP address. Broadcast packets are wasteful in that they wake up every host on a link, even if that host does not provide the relevant service. Rather than using broadcast addresses as a way for a host to find a service, IPv6 uses a well-known multicast address for each service being offered. Hosts that are prepared to provide a service register to listen on the well-known multicast address for that service.

When an IPv6 address is written, it is represented as a set of colon-separated hex bytes. The value between each set of colons represents a 16-bit value. For example, the string:

1080:0:0:8:0:0:200C:417A

represents a unicast address in the IPv6 network. When written out as text, a portion of the address that contains zeros may be abbreviated with a double colon:

1080::8:0:0:200C:417A

The first set of two zeros was eliminated in this particular address. When an address is abbreviated, only one set of zeros may be abbreviated. For the run of zeros being eliminated, either all zeros must be removed or none. The following are examples of improper abbreviations of the preceding address:

1080::0:8:0:0:200C:417A
1080::8::200C:417A

The first does not subsume the entire first set of zeros. The second is ambiguous because you cannot tell how to divide the four zeros between the two areas marked with double colons.

Unicast and multicast addresses are differentiated by the bits set at the beginning of the address. All unicast addresses begin with the bits 100, and multicast addresses start with 1111 1111. Examples of the most common addresses are shown in Table 13.5. The unspecified address is used by a host that has not yet been assigned an address when it is in the process of bringing up its network interface. The solicited-node address is used during neighbor discovery, which is covered later in this section.

Table 13.5. Well-known IPv6 addresses.

Address

Description

FF02::1

all nodes multicast address (link local)

FF02::2

all routers multicast address (link local)

FF05::2

all routers multicast address (site local)

FF02:0:0:0:0:1:FF00::/104

solicited node address

::1

the loopback address

::

the unspecified address

A piece of baggage that was not carried over from IPv4 to IPv6 was the concept of network classes in addresses. IPv6 always uses the CIDR style of marking the boundary between the network prefix (hereafter referred to simply as the prefix) and the interface identifier, which is what identifies an interface on a particular host. The following examples all define the same network, which has a 60-bit prefix:

1234:0000:0000:1230:0000:0000:0000:0000/60
1234::1230:0:0:0:0/60
1234:0:0:1230::/60

IPv6 Packet Formats

When IPv6 was being designed, one goal was to reduce the amount of work necessary for a router to forward a packet. This reduction was addressed as follows:

  • Simplification of the packet header. Comparing the IPv6 packet header in Figure 13.12 against the IPv4 header shown in Figure 13.4, we see that there are four fewer fields in the IPv6 header and that only one of them needs to be modified while the packet is in transit: the hop limit. The hop limit is decremented every time the packet is forwarded by a router until the hop limit reaches 0 when the packet is dropped.

    Figure 13.12. IPv6 packet header.

    [View full size image]

  • The packet header is a fixed size. The IPv6 header never carries any options or padding within it. Options processing in IPv4 is an expensive operation that must be carried out whenever an IPv4 packet is sent, forwarded, or received.

  • IPv6 strongly discourages the use of fragmentation at the network layer. Avoiding packet fragmentation simplifies packet forwarding as well as processing by hosts (hosts are where the reassembly of fragmented packets take place).

All these simplifications make processing IPv6 packets less compute-intensive than processing those of IPv4. Completely removing features that were inconvenient, such as options or fragmentation, would have decreased the acceptance of IPv6. Instead, the designers came up with a way to add these features, and several others, without polluting the base packet header. Extra features and upper-layer protocols in IPv6 are handled by extension headers. An example packet is shown in Figure 13.13. All extension headers begin with a next-header field as well as an 8-bit length field that shows the length of the extension in units of 8 bytes. All packets are aligned to an 8-byte boundary. The IPv6 header and the extension headers form a chain linked together by the next-header field present in each of them. The next-header field identifies the type of data immediately following the header that is currently being processed and is a direct descendant of the protocol field in IPv4 packets. TCP packets are indicated by the same number in both fields (6). Routers do not look at any of the extension headers when forwarding packets except for the hop-by-hop options header that is meant for use by routers. Each of the extension headers also encodes its length in some way. TCP packets are unaware of being carried over IPv6 and use their original packet-header format, which means they carry neither a next-header field nor a length. The length is computed as it is in IPv4.

Figure 13.13. Extension headers. Key: AH—authentication header (type 51); ESP—encapsulating-security payload (type 50).

Hosts are required to encode and decode extension headers in a particular order so that it is unnecessary to ever backtrack through a packet. The order in which headers should appear is shown in Figure 13.13. The hop-by-hop header (type 0) must immediately follow the IP header so that routers can find it easily. The authentication header (AH) and encapsulating-security payload (ESP) headers are used by security protocols that are discussed in Section 13.10 and must come before the TCP header and data, since the information in the security headers must be retrieved before they can be used to authenticate and decrypt the TCP header and data.

Changes to the Socket API

It has always been the policy of the Internet Engineering Task Force (IETF) to specify protocols and not implementations. For IPv6, this rule was bent so that application developers would have an API to which they could code and speed the migration of applications to IPv6. The designers took the original sockets interface, as it was then implemented in BSD, and specified extensions [Gilligan et al., 1999] that are included in FreeBSD. There were several goals in extending the sockets API:

  • The changes should not break existing applications. The kernel should provide backward compatibility for both source and binary.

  • Minimize the number of changes needed to get IPv6 applications up and running.

  • Ensure interoperability between IPv6 and IPv4 hosts.

  • Addresses carried in data structures should be 64-bit aligned to obtain optimum performance on 64-bit architectures.

Adding a new address type was easy because all the routines that handle addresses, such as bind, accept, connect, sendto, and recvfrom, already work with addresses as opaque entities. A new data structure, sockaddr_in6, was defined to hold information about IPv6 endpoints as shown in Figure 13.14. The sockaddr_in6 structure is similar to the sockaddr_in shown in Section 11.4. It contains the length of the structure, the family (which is always AF_INET6), a 16-bit port that identifies the transport-layer endpoint, a flow identifier, network-layer address, and a scope identifier. Many proposals have been put forth for the use of the flow information and scope identifier, but these fields are currently unused. The flow information is intended as a way of requesting special handling for packets within the network. For example a real-time audio stream might have a particular flow label so that it would be given priority over less time-critical traffic. Although the idea is simple to explain, its implementation in a network where no one entity controls all the equipment is problematic. At present there is no way to coordinate what a flow label means when it leaves one network and enters another. Until this conundrum is solved, the flow label will be used only in private network deployments and research labs.

Figure 13.14. IPv6-domain socket-address structure.

IPv6 defines several scopes in which an address can be used. In IPv4, all addresses were global in scope, meaning that they were valid no matter where they were found on the Internet. The defined scopes in IPv6 are link local, site local, organization local, and global. An address in a lesser scope may not be passed out to a broader scope. For example a link-local address will not be forwarded to another link by a router.

Working with 128-bit addresses by hand is clumsy and error prone. Applications are expected to deal almost exclusively with named entities for IPv6 by using the domain name system (DNS) [Thomson & Huitema, 1995]. The original API for looking up an address from a hostname, gethostbyname(), was specific to the AF_INET domain, so a new API was added to lookup an IPv6 address given a name. When a client wishes to find a server, it uses the gethostbyname2() routine that takes as one of its arguments an address family.

struct hostent * gethostbyname2(
        char *name,
        int address_family);

This new API can work with either IPv4 or IPv6 addresses because the second argument specifies the address family and the return value is a pointer to a structure that contains the address type being returned.

Autoconfiguration

A goal of IPv6 is to make adding a computer to the network a simpler process. The mechanisms and protocols that are used to reach this goal are called autoconfiguration. For a host to be automatically configured it has to be able to discover several pieces of information from the network without any prior knowledge. The host must be able to automatically figure out its own address, the address of its next hop router, and the network prefix of the link to which it is attached. To communicate with other hosts on its link and with its next hop router, a host needs the link-level addresses for those other systems. These questions are answered by the neighbor-discovery protocol that is a part of IPv6 and is defined in Narten et al. [1998]. Neighbor discovery either enhances or replaces disparate protocols that were a part of IPv4 and unifies them in a set of ICMPv6 messages [Conta & Deering, 1998]. Because neighbor discovery uses ICMPv6 and thus runs above the IPv6 layer, as long as IPv6 runs on a particular link type, so will neighbor discovery and its auto-configuration services. We will look at two aspects of the neighbor-discovery protocol. The first will be router discovery—how a node finds its next hop router—and the second will be neighbor discovery itself.

A host finds its next hop router in two different ways. IPv6 routers periodically send router-advertisement messages to all nodes multicast address. The format of a router-advertisement message is shown in Figure 13.15. All hosts configured to pick up these multicast packets will see the router advertisement and process it. Although router advertisements are sent often enough to make sure that all hosts on a link know the location of their router and know when it has failed, this mechanism is insufficient for bringing a new host up on the link. When a host first connects to the network, it sends a router-solicitation message to the multicast address of all routers. A router that receives a valid solicitation must immediately send a router advertisement in response. The advertisement will be sent to the multicast address of all nodes unless the router knows that it can successfully send a unicast response to the host that sent the solicitation. A router may send an option with the advertisement that includes the link-layer address of the router. If the link-layer address option is included, the receiving hosts will not need to do neighbor discovery before sending packets to the router.

Figure 13.15. Router advertisement. Key: M—managed flag; O—other flag.

[View full size image]

Each host maintains a linked list of its router entries. A single router entry is shown in Figure 13.16. Whenever a router advertisement is received, it is passed to the defrtrlist_update() routine that checks the message to see if it represents a new router and, if so, places a new entry at the head of the default router list. Each router advertisement message contains a lifetime field. This lifetime controls how long an entry may stay in the default router list. Whenever defrtrlist_update() receives a router advertisement for a router that is already present in the default router list, that router's expiration time is extended.

Figure 13.16. Router entry.

For a host to determine if the next hop to which a packet should be sent is on the same link as itself, it must know the prefix for the link. Historically, the prefix was manually configured on each interface in the system, but now it is handled as part of router discovery.

Prefix information is sent as an option within a router advertisement. The format of the prefix option is shown in Figure 13.17. Each prefix option carries a 128-bit address. The number of valid bits in this address is given by the prefix-length field of the option. For example, the prefix given in the preceding example would be sent in a prefix option with

1234:0000:0000:1230:0000:0000:0000:0000

Figure 13.17. Prefix option. Key: O—onlink flag; A—auto flag.

[View full size image]

encoded into the prefix field and 60 stored in the prefix-length field. Each prefix is only valid for the period shown by the valid lifetime. Later router advertisements that contain prefix options will have valid lifetimes that move into the future. When a host discovers that it has a prefix whose lifetime has expired, the prefix is removed from the interface with which it is associated, and the expired prefix no longer determines whether a destination address is on the local link.

When a host has a packet for another host on its link, including its next-hop router, it must find the link-layer address of the host to which it wishes to send the packet. In IPv4, this process was handled by the address-resolution protocol (ARP); see Section 12.8. A problem with ARP is that it is Ethernet specific and has encoded in it assumptions about link-layer addresses that makes it difficult to adapt to other link types.

A host learns the link-layer addresses of its neighbors using a pair of messages: the neighbor solicitation and the neighbor advertisement. When the kernel wants to send an IPv6 packet to another host, the packet eventually passes through the ip6_output() routine, which does various checks on the packet to make sure that it is suitable for transmission. All properly formed packets are then passed down to the neighbor-discovery module via the nd6_output() routine that handles mapping an IPv6 address to a link-layer address. Once the packet has a correct link-layer destination address, it is passed to a network-interface driver via the driver's if_output() routine. The relationships between the various protocol modules are shown in Figure 13.18 (on page 568). The neighbor-discovery module does not have an nd_input() routine because it receives messages via the ICMPv6 module. This inversion of the protocol layering allows the neighbor-discovery protocol to be independent of the link layer. In IPv4, the ARP module is hooked into the network interface so that it can send and receive messages. The connection between ARP and the underlying link-layer interfaces means that the ARP code must understand every link type that the system supports.

Figure 13.18. IPv6-module relationships.

Link-layer addresses are stored in the routing table, and that is where nd6_output() attempts to lookup the link-layer address for the packets that are passed to it. When the host does not yet know the link-layer address for the destination, the outgoing packet must be held until neighbor discovery completes. Each rtentry structure contains a pointer to link-layer information. When a route entry stores information for IPv6, it points to an llinfo_nd6 structure, as shown in Figure 13.19. Before a neighbor solicitation is sent, the packet that the kernel is attempting to transmit is saved by making the In_hold field of the llinfo_nd6 structure point to it. It is only possible to store one packet at a time, so if the system attempts to transmit another packet to the same link-layer destination before a neighbor advertisement is received, the first packet will be lost and upper layers will have to retransmit it. The nd6_output() routine does not wait for the neighbor advertisement but returns. When a response is received as a neighbor advertisement, it is processed by the IPv6 and ICMPv6 modules and is finally passed into the neighbor-discovery module by a call to the nd6_na_input() routine, as shown in Figure 13.18. The nd6_na_input() routine records the link-layer address and checks to see if a packet was being held for transmission to that destination. If there is a packet awaiting transmission, the nd6_output() routine is invoked with the saved packet. A link-layer address for the saved packet's destination is now in the system, so nd6_output() will copy the link-layer address into the mbuf chain and invoke the network interface's if_output() routine to transmit the packet.

Figure 13.19. Neighbor-discovery link-layer information.

Once per second the nd6_timer() routine walks the neighbor-discovery link-layer address list as well as the default-router and interface lists and removes the entries that have passed their expiration time. Removing stale entries prevents the system from trying to send data to a host that has failed or become unreachable.


13.10. Security

We mentioned in Section 13.9 that a suite of security protocols were developed as part of IPv6. These protocols were written to be independent of a particular version of IP, so they have been integrated into IPv4 and IPv6. At the network layer, security mechanisms have been added to provide authentication so that one host can know with whom it is communicating. Encryption has been added so that data can be hidden from untrusted entities as it crosses the Internet. The protocols that collectively provide security within the network layer are referred to as IPSec.

Placing the security protocols at the network layer within the protocol stack was not an arbitrary decision. It is possible to place security at just about any layer within a communication system. For example, the secure sockets layer (SSL) supports communication security at the application layer and allows a client and a server to communicate securely over an arbitrary network. At the opposite end of the spectrum are the various protocols that support security over wireless networks that work at the data-link layer. The decision to put security at the network layer was made for several reasons:

  • The IP protocols act as a uniform platform into which to place the security protocols. Differences in underlying hardware, such as different types of network media, did not have to be taken into account when designing and implementing IPSec because if a piece of hardware could send and receive IP datagrams, then it could also support IPSec.

  • Users need not do any work to use the security protocols. Because IPSec is implemented at the network, instead of the application layer, users that run network programs are automatically working securely as long as their systems administrators have properly configured the system.

  • Key management can be handled in an automatic way by system daemons. The hardest problem in deploying network security protocols is giving out and canceling the keys used to encrypt the data. Since IPSec is handled in the kernel, and is not usually dealt with by users, it is possible to write daemons to handle the management of keys.

Security within the context of IPSec means several things:

  • The ability to trust that a host is who it claims to be (authentication)

  • Protection against the replay of old data

  • Confidentiality of data (encryption)

Providing a security architecture for the Internet protocols is a complex problem. The relevant protocols are covered in several RFCs, and an overview is given in Kent & Atkinson [1998a].

FreeBSD contains two implementations of IPSec. One is derived from the KAME code base, and the other, known as Fast IPSec, is a reworking of the KAME code base so that it can work with the OpenBSD cryptographic subsystem [Leffler, 2003a]. The largest difference between the two code bases is that the Fast IPSec code does not have any cryptographic algorithms built into it and depends wholly on the cryptographic subsystem to handle the work of encrypting, decrypting, and otherwise manipulating data. Although the KAME code was the first implementation available of the IPSec protocols, and is still the most widely used, we discuss the Fast IPSec code because it allows us to explain the hardware cryptography subsystem that has been added to FreeBSD.

IPSec Overview

The protocols that make up IPSec provide a security framework for use by hosts and routers on the Internet. Security services, such as authentication and encryption, are available between two hosts, a host and a router, or two routers. When any two entities on the network (hosts or routers) are using IPSec for secure communication, they are said to have a security association (SA) between them. Each SA is unidirectional, which means that traffic is only secured between two points in the direction in which the SA has been set up. For a completely secure link two SAs are required, one in each direction.

SAs are uniquely identified by their destination address, the security protocol being used, and a security-parameter index (SPI), which is a 32-bit value that distinguishes among multiple SAs terminating at the same host or router. The SPI is the key used to lookup relevant information in the security-association database that is maintained by each system running IPSec.

An SA can be used in two modes. In transport mode, a portion of the IP header is protected as well as the IPSec header and the data. The IP header is only partially protected because it must be inspected by intermediate routers along the path between two hosts, and it is not possible, or desirable, to require every possible router to run the IPSec protocols. One reason to run security protocols end to end is so intermediate routers do not have to be trusted with the data they are handling. Another reason is that security protocols are often computationally expensive and intermediate routers often do not have the computational power to decrypt and reencrypt every packet before it is forwarded.

Since only a part of the IP header is protected in transport mode, this type of SA only provides protection to upper-layer protocols, those that are completely encapsulated within the data section of the packet, such as UDP and TCP. Figure 13.20 shows a transport-mode SA from Host A to Host D as well as the packet that would result. Host A sets up a normal IP packet with a destination of host D. It then adds the IPSec header and data. Finally, it applies whatever security protocol has been selected by the user and sends the packet, which travels through Router B to Router C and finally to Host D. Host D decrypts the packet by looking up the security protocol and keys in its security-association database.

Figure 13.20. Security association in transport mode. Key: AH—authentication header; ESP—encapsulating-security payload; SPI—security-parameter index.

The other mode is tunnel mode, shown in Figure 13.21 (on page 572), where the entire packet is placed within an IP-over-IP tunnel [Simpson, 1995]. In tunneling, the entire packet, including all the headers and data, are placed as data within another packet and sent between two locations. Host A wants to send a packet to Host D. When the packet reaches Router B, it is placed in a secure tunnel between Router B and Router C. The entire original packet is placed inside a new packet and secured. The outer IP header identifies only the endpoints of the tunnel (Router B and Router C) and does not give away any of the original packet's header information. When the packet reaches the end of the tunnel at Router C, it is decrypted and then sent on to its original destination of Host D. In this example, neither Host A nor Host D knows that the data have been encrypted nor do they have to be running the IPSec protocols to participate in this secure communication.

Figure 13.21. Security association in tunnel mode. Key: AH—authentication header; ESP—encapsulating-security payload; SPI—security-parameter index.

Tunnel mode is only used for host-to-router or router-to-router communications and is most often seen in the implementation of virtual private networks that connect two private networks or connect users to a corporate LAN over the public Internet.

Security Protocols

There are two security protocols specified for use with IPSec: the authentication header (AH) and the encapsulating-security payload (ESP), each of which provides different security services [Kent & Atkinson, 1998b; Kent & Atkinson, 1998c]. Both protocols are used with IPv4 and IPv6 without changes to their headers. This dual usage is possible because the packet headers are really IPv6 extension headers that properly encode information about the other protocols following them in the packet.

The AH protocol provides a packet-based authentication service as well as protection against an attacker attempting to replay old data. To understand how AH provides security, it is easiest to look at its packet header, shown in Figure 13.22. The next-header field identifies the type of packet that follows the current header. The next-header field uses the same value as the one that appears in the protocol field of an IPv4 packet: 6 for TCP, 17 for UDP, and 1 for ICMP. The payload length specifies the number of 32-bit words that are contained in the authentication header minus 2. The fudge factor of removing 2 from this number comes from the specification for IPv6 extension headers. The SPI was just explained and is simply a 32-bit number that is used by each endpoint to lookup relevant information about the security association.

Figure 13.22. Authentication header.

[View full size image]

Authentication is provided by computing an integrity-check value (ICV) over the packet. If an AH is used in transport mode, then only parts of the IP header are protected because some of the fields are modified by intermediate routers in transit and the changes are not predictable at the sender. In tunnel mode, the whole header is protected because it is encapsulated in another packet, and the ICV is computed over the original packet. The ICV is computed using the algorithm specified by the SPI with the result stored in the authentication-data field of the authentication header. The receiver uses the same algorithm, requested by the SPI to compute the ICV on the packet it received, and compares this value with the one found in the packet's authentication-data field. If the values are the same, then the packet is accepted; otherwise, it is discarded.

One possible attack on a communication channel is to send new or false data as if it were coming from the authentic source, which is called a replay attack. To guard against a replay attack, the AH protocol uses a sequence-number field to uniquely identify each packet that is transmitted across an SA. This sequencenumber field is distinct from the field of the same name in TCP. When an SA is established, both the sender and receiver set the sequence number to zero. The sender increments the sequence number before transmitting a packet. The receiver implements a fixed-size sliding window, with its left edge being the lowest sequence number that it has seen and validated and the right edge being the highest. When a new packet is received, its sequence number is checked against the window with three possible results:

  • The packet's sequence number is less than the one on the left edge of the window, and the packet is discarded.

  • The packet's sequence number is within the window. The packet is checked to see if it is a duplicate, and if so is discarded. If the packet is not a duplicate, it is inserted into the window.

  • The packet's sequence number is to the right of the current window. The ICV is verified, and, if correct, the window is moved to the right to encompass the new sequence number value.

When the sequence number rolls over, after over 4 billion packets, the security association must be torn down and restarted. This restart is only a slight inconvenience because at gigabit Ethernet rates of 83,000 packets per second it takes over 14 hours for the security sequence number to roll over.

All senders assume that a receiver is using the antireplay service and always increment the sequence number, but it is not required for the receiver to implement the antireplay service, and it may be turned off at the discretion of the operator of the receiving system.

In addition to the services provided by the AH, the ESP also provides confidentiality using encryption. As with the AH it is easiest to understand the ESP if we examine its packet header, shown in Figure 13.23. The ESP header contains all the same fields as were found in the AH header, but it adds three more. The encrypted data sent using an ESP is stored in the payload-data field of the packet. The padding field that follows the payload data may be used for three purposes:

  • The encryption algorithm might require that the data to be encrypted be some multiple number of bytes. The padding data is added to the data to be encrypted so that the chunk of data is of the correct size.

  • Padding might be required to properly align some part of the packet. For example, the pad-length and next-header fields must be right-aligned in the packet, and the authentication-data field must be aligned on a 4-byte boundary.

  • The padding may also be used to obscure the original size of the payload in an attempt to prevent an attacker from gaining information by watching the traffic flow.

Figure 13.23. Encapsulating security-protocol header.

[View full size image]

Key Management

User-level applications cannot use IPSec in the same way that they use transport protocols like UDP and TCP. For example, an application cannot open a secure socket to another endpoint using IPSec. Instead, all SAs are kept in the kernel and managed using a new domain and protocol family called PF_KEY_V2 [McDonald et al., 1998].

The automated distribution of keys for use in IPSec is handled by the Internet key exchange (IKE) protocol [Harkins & Carrel, 1998]. User-level daemons that implement the IKE protocol, such as Racoon, interact with the kernel using PF_KEY_V2 sockets [Sakane, 2001]. As these daemons are not implemented in the kernel, they are beyond the scope of this book.

User-level applications interact with the security database by opening a socket of type PF_KEY. There is no corresponding AF_KEY address family. Key sockets are based on the routing-socket implementation and function much like a routing socket. Whereas the routing-socket API manipulates the kernel routing table, the key-socket API manages security associations and policies. Key sockets support a connectionless-datagram facility between user applications and the kernel. User-level applications send commands in packets to the kernel's security database. Applications can also receive messages about changes to the security database, such as the expiration of security associations, by reading from a key socket.

The messages that can be sent using a key socket are shown in Table 13.6. Two groups of messages are defined for key sockets: a base set of messages that all start with SADB and a set of extension messages that starts with SADB_X. The type of the message is the second part of the name. In FreeBSD, the extension messages manipulate a security-policy database (SPDB) that is separate from the security-association database (SADB).

Table 13.6. PF_KEY messages.

Message type

Description

SADB_GETSPI

retrieve a unique security index from the kernel

SADB_UPDATE

update an existing security association

SADB_ADD

add a new security association with a known security index

SADB_DELETE

delete an existing security association

SADB_GET

retrieve information on a security association

SADB_ACQUIRE

sent to user-level daemons when kernel needs more information

SADB_REGISTER

tell the kernel this application can supply security information

SADB_EXPIRE

sent from the kernel to the application when an SA expires

SADB_FLUSH

tell the kernel to flush all SAs of a particular type

SADB_DUMP

tell the kernel to dump all SA information to the calling socket

SADB_X_PROMISC

this application wants to see all PF_KEY messages

SADB_X_PCHANGE

message sent to passive listeners

SADB_X_SPDUPDATE

update the security policy database (SPDB)

SADB_X_SPDADD

add an entry to the SPDB

SADB_X_SPDDELETE

delete an entry from the SPDB by policy index

SADB_X_SPDGET

get an entry from the SPDB

SADB_X_SPDACQUIRE

message sent by kernel to acquire an SA and policy

SADB_X_SPDDUMP

tell the kernel to dump its policy database to the calling socket

SADB_X_SPDFLUSH

flush the policy database

SADB_X_SPDSETIDX

add an entry in the SPDB by its policy index

SADB_X_SPDEXPIRE

tell the listening socket that an SPDB entry has expired

SADB_X_SPDDELETE2

delete an SPDB entry by policy identifier

Key: SA — security association; SADB — security-association database; SPDB — security-policy database.

Key-socket messages are made up of a base header, shown in Figure 13.24, and a set of extension headers. The base header contains information that is common to all messages. The version ensures that the application will work with the version of the key-socket module in the kernel. The command being sent is encoded in the message-type field. Errors are sent to the calling socket using the same set of headers that are used to send down commands. Applications cannot depend on all errors being returned by a send or write system call made on the socket, and they must check the error number of any returned message on the socket for proper error handling. The errno field is set to an appropriate error number before the message is sent to the listening socket. The type of security association that the application wants to manipulate is placed in the SA-type field of the packet. The length of the entire message, including the base header, all extension headers, and any padding that has been inserted is stored in the length field. Each message is uniquely identified by its sequence and PID fields that match responses to requests. When the kernel sends a message to a listening process the PID is set to 0.

Figure 13.24. PF_KEY base header.

[View full size image]

The security-association database and security-policy database cannot be changed using only the base header. To make changes, the application adds one or more extension headers to its message. Each extension header begins with a length and a type so that the entire message can be easily traversed by the kernel or an application. An association extension is shown in Figure 13.25. The association extension makes changes to a single security association, such as specifying the authentication or encryption algorithm to be used.

Figure 13.25. PF_KEY association extension.

[View full size image]

Whenever an association extension is used, an address extension must be present as well, since each security association is identified by the network addresses of the communicating endpoints. An address extension, shown in Figure 13.26, stores information on the IPv4 or IPv6 addresses using sockaddr structures.

Figure 13.26. PF_KEY address extension.

[View full size image]

One problem with the current PF_KEY implementation is that it is a datagram protocol, and the message size is limited to 64 kilobytes. A 64-kilobyte limit is not important to users with small databases, but when a system using IPSec is deployed in a large enterprise, with hundreds and possibly thousands of simultaneous security associations, the SADB will grow large, and this limitation makes it more difficult to write user-level daemons to manage the kernel's security databases.

The purpose of key sockets is to manage the security-association database stored in the kernel. Like many other data structures in FreeBSD, security-association structures are really objects implemented in C. Each security-association structure contains all the data related to a specific security association as well as the set of functions necessary to operate on packets associated with it.

The security-association database is stored as a doubly linked list of security-association structures. A security-association structure is shown in Figure 13.27 (on page 578). Each security association can be shared by more than one entity in the system, which is why they contain a reference count. Security associations can be in four states: LARVAL, MATURE, DYING, and DEAD. When an SA is first being created, it is put into the LARVAL state, which indicates that it is not currently usable but is still being set up. Once an SA is usable, it moves to the MATURE state. An SA remains in the MATURE state until some event, such as the SA exceeding its lifetime, moves it to the DYING state. SAs in the DYING state can be revived if an application makes a request to use an SA with the same parameters before it is marked as DEAD.

Figure 13.27. Security-association structure.

[View full size image]

The security-association structure contains all the information on a particular SA including the algorithms used, the SPI, and the key data. All this information is used in processing packets for a particular association. The lifetime fields limit the usage of a particular SA. Although an SA is not required to have a lifetime, and so might not expire, recommended practice is to set a lifetime. Lifetimes can be given a time limit using the addtime and usetime fields. Lifetimes can be given a data-processing limit using the bytes field. The three lifetime structures pointed to by the security association encode the current usage for the association as well as its hard and soft limits. When reached, the soft-lifetime value puts the SA into the DYING state to show that its useful life is about to end. When reached, the hard-lifetime value indicates that the SA is no longer usable at all. Once an SA passes the hard-lifetime limit, it is set to the DEAD state and can be reclaimed. The current-lifetime structure contains the present usage values for the SA—for example, how many bytes have been processed since the SA was created.

Each security-association structure has several tables of functions that point to routines that do the work on packets handled by that association. The tdb_xform table contains pointers to functions that implement the initialization and input and output functions for a particular security protocol such as ESP or AH. The other three tables are specific to a protocol and contain pointers to the appropriate cryptographic functions for handling the protocol being used by the SA. The reason for having this plethora of tables is that the cryptographic subsystem ported from OpenBSD used these tables to encapsulate the functions that do the real work of cryptography. To simplify the maintenance of the code, this set of interfaces and tables was retained during the port. A useful side effect of having these tables is that it makes adding new protocols or cryptographic routines simple. We describe how these tables are used later in this section.

Key sockets are implemented in the same way as other socket types. There is a domain structure, keydomain; a protocol-switch structure, keysw; a set of user-request routines, key_usrreqs; and an output-routine, key_output(). Only those routines necessary for a connectionless-datagram type of protocol are implemented in the key_usrreqs structure. Any attempt to use a key socket in a connection-oriented way—for instance, by calling connect on a key socket—will result in the kernel returning EINVAL to the caller.

When an application writes to a key socket, the message is eventually transferred down into the kernel and is handled by the key_output() routine. After some rudimentary error checking the message is passed to key_parse(), which does more error checks, and then is finally shuttled off through a function-pointer switch called key_types. The functions pointed to by key_types are those that do the manipulation of the security-association and security-policy databases.

If the kernel needs to send a message to listening applications because of changes in the security databases, it uses the key_sendup_mbuf() routine to copy the message to one or more listening sockets. Each socket receives its own copy of the message.

IPSec Implementation

The IPSec protocols affect all areas of packet handling in the IPv4 and IPv6 protocol stacks. In some places, IPSec uses the existing networking framework, and in others, direct callouts are made to do some part of the security processing. We will look at three of the possible paths through the IPv4 stack: inbound, outbound, and forwarding.

One twist that IPSec adds to normal packet processing is the need to process some packets more than once. An example is the arrival of an encrypted packet bound for the current system. The packet will be processed once in its encrypted form and then a second time, by the same routines, after it has been decrypted. This multipass processing is unlike regular TCP or UDP processing where the IP header is stripped from the packet and the result is handed to the TCP or UDP modules for processing and eventual delivery to a socket. This continuation style of processing packets is one reason that the IPSec software makes extensive use of packet tags. Another reason to use packet tags is that parts of IPSec, namely the cryptographic algorithms, can be supported by special-purpose hardware accelerators. A hardware accelerator may do all or part of the security processing, such as checking a packet's authentication information or decrypting the packet payload, and then pass the resulting packet into the protocol stack for final delivery to a waiting socket. The hardware needs some way to tell the protocol stack that it has completed the necessary work. It is neither possible, nor desirable, to store this information in the headers or data of the packet. Adding such information to a packet's header is an obvious security hole because a malicious sender could simply set the appropriate field and bypass the security processing. It would have been possible to extend the mbuf structure to handle this functionality, but packet tags are a more flexible way of adding meta-data to packets without modifying a key data-structure in the network stack. The tags used by IPSec are described in Table 13.7.

Table 13.7. IPSec packet tags.

Tag

Description

IPSEC_IN_DONE

Inbound IPSec processing complete.

IPSEC_OUT_DONE

Outbound IPSec processing complete.

IPSEC_IN_CRYPTO_DONE

Inbound IPSec processing handled by hardware.

IPSEC_OUT_CRYPTO_DONE

Outbound IPSec processing handled by hardware.

As we saw in Section 13.3, when an IPv4 packet is received by the kernel it is initially processed by ip_input(). The ip_input() routine does two checks on packets that are related to IPSec. The first is to see if the packet is really part of a tunnel. If a packet is being tunneled, then if it has been processed by the IPSec software already, it can bypass any filtering by filter hooks or the kernel fire-walling code. The second check is done when a packet is to be forwarded. Routers can implement security policies on packets that are forwarded. Before a packet is passed to ip_forward(), it is checked by calling the ipsec_getpolicy() function to see if there is a policy that is associated with the packet itself. The ipsec_getpolicybyaddr() function is called to check if there is a policy associated with the address of the packet. If either function returns a pointer to a policy routine, the packet is passed to that policy routine to be checked. If the packet is rejected, it is silently dropped and no error is returned to the sender.

When ip_input() has determined that the packet is valid and is destined for the local machine, the protocol-stack framework takes over. The packet is passed to the appropriate input routine using the pr_input field of the inetsw structure. Although packets using different protocols have different entry points, they eventually wind up being passed to a single routine, ipsec_common_input(), for processing. The ipsec_common_input() routine attempts to find the appropriate security-association structure for the packet based on its destination address, the security protocol it is using, and the SPI. If an appropriate association is found, then control is passed to the input routine contained in the SA's xform-switch structure. The security protocol's input routine extracts all the relevant data from the packet—for example, the key being used—and creates a cryptography-operation descriptor. This descriptor is then passed into the cryptographic routines. When the cryptographic routines have completed their work, they call a protocol-specific callback routine, which modifies the mbufs associated with the packet so that it may now be passed, unencrypted, back into the protocol stack via the ip_input() routine.

Applications do not know that they are using IPSec to communicate with other hosts in the Internet. For outbound packets, the use of IPSec is really controlled from within the ip_output() routine. When an outbound packet reaches the ip_output() routine, a check is made to see if there is a security policy that applies to the packet, either because of its destination address or because of the socket that sent it. If a security policy is found, then the packet is passed into the IPSec code via the ipsec4_process_packet() routine. If a security association has not been set up for this particular destination, one is created in the security-association database for it. The ipsec4_process_packet() uses the output() routine from the xform switch in the security association to pass off the packet to the security-protocol's output routine. The security-protocol's output routine uses the appropriate cryptographic routine to modify the packet for transmission. Once the packet has been modified appropriately, it is passed again into ip_output() but with the tag PACKET_TAG_IPSEC_OUT_DONE attached to it. This tag marks the packet as having completed IPSec processing, showing that it can now be transmitted like any other packet.

Cryptography Subsystem

Underlying all the security protocols provided by IPSec is a set of APIs and libraries that support cryptography. The cryptographic subsystem in FreeBSD supports both symmetric and asymmetric cryptography. Symmetric cryptography, used by IPSec, uses the same key to encrypt data as it does to decrypt it. Asymmetric cryptography, which implements public key encryption, uses one key to encrypt data and another key to decrypt it. This section describes how symmetric cryptography is implemented as it relates to a specific client, IPSec.

The cryptographic subsystem was ported from OpenBSD and optimized for a fully preemptive, SMP kernel [Leffler, 2003b]. In FreeBSD, cryptographic algorithms exist either in software or special-purpose hardware. The software module that provides support for cryptography is implemented in exactly the same way as the drivers for cryptographic hardware. This similarity means that, from the cryptography subsystem's point of view, the software and hardware drivers are the same. Upper-level users of the cryptography subsystem, such as IPSec, are all presented with the same API whether the cryptographic operations they request are being done in hardware or software.

The cryptography subsystem is implemented by two sets of APIs and two kernel threads. One set of APIs is used by software that wishes to use cryptography; the other set is used by device-driver writers to provide an interface to their hardware. The model of computation supported by the cryptographic subsystem is one of job submission and callbacks where users submit work to be done to a queue and supply a pointer to a function that will be called when the job is completed.

Before a cryptography user can submit a work to the cryptography subsystem, they must first create a session. A session is a way of encapsulating information about the type of work that the user is requesting. It is also a way of controlling the amount of resources consumed on the device, since some devices have a limitation to the amount of concurrent work they can support. A user creates a session using the crypto_newsession() routine that returns either a valid session identifier or an error.

Once the user has a proper session identifier, they then request a cryptographic descriptor, shown in Figure 13.28. The user fills in the fields of the cryptographic descriptor, including supplying an appropriate callback in the crp_callback element. When the descriptor is ready, it is handed to the cryptographic subsystem via the crypto_dispatch() routine that puts it on a queue to be processed. When the work is complete, the callback is invoked. All callbacks are of the form:

int (*crp_callback)(
    struct cryptop *arg);

Figure 13.28. Cryptographic descriptor.

If an error has occurred, the error code is contained in the crp_etype field of the cryptographic descriptor that is passed to the callback.

A set of device drivers provides the low-level interface to specialized cryptographic hardware. Each driver provides three function pointers to the cryptographic subsystem when it registers itself. Driver registration is done via a call to the crypto_register() routine.

crypto_register(
    u_int32_t driverid,
    int alg,
    u_intl6_t maxoplen,
    u_int32_t flags,
    int (*newsession)(void*, u_int32_t*, struct cryptoini*),
    int (*freesession)(void*, u_int64_t),
    int (*process)(void*, struct cryptop *, int),
    void *arg);

The newsession() routine is called by the cryptographic subsystem whenever the crypto_newsession () routine is called by a user. The freesession() routine is called whenever the crypto_freesession() routine is called by a user, and the process() routine is called by the crypto_proc() kernel thread to pass operations into the device.

The lower half of the cryptographic subsystem uses two software interrupt threads and two queues to control the underlying hardware. Whenever there are requests on the crp_q queue, the crypto_proc() thread dequeues them and sends them to the underlying device, using the crypto_invoke() routine. Once invoked, the underlying hardware has the responsibility to handle the request. The only requirement is that when the hardware has completed its work, the device driver associated with the hardware must invoke crypto_done() that either enqueues the callback on the crp_ret_q queue or more rarely directly calls the user's callback. The crp_ret_q queue is provided because the crypto_done() routine often will be called from an interrupt context, and running the user's callback with interrupts locked out will degrade the interactive performance of the system. When running in an interrupt context, the callback will be queued and then handled later by the crypto_ret_proc software interrupt thread. This use of queues and software interrupt threads effectively decouples the kernel from any possible performance issues introduced by a variety of cryptographic hardware.

Unfortunately, there are several problems with the system just described:

  • Using multiple threads requires two context switches per cryptographic operation. The context switches are nontrivial and severely degrade throughput.

  • Some callback routines do little work, and so moving all callbacks out of the device driver's interrupt service routine adds another context switch that is expensive and unnecessary.

  • The dispatch queue batches operations, but many users of the cryptographic subsystem, including IPSec, do not batch operations, so this shunting of work into the dispatch queue is unnecessary overhead.

To address these performance problems several changes were made to the cryptographic subsystem. Cryptographic drivers are now supplied a hint whether there is more work to follow when work is submitted to them. The drivers can decided whether to batch work based on this hint and, where requests are not batched, completely bypass the crp_q queue. Cryptographic requests whose callback routines are short mark their requests so that the underlying device executes them directly instead of queueing them on the crypto_ret_q queue. The optimization of bypassing the crypto_req_q queue is especially useful to users of the /dev/crypto device, whose callback routine awakens only the thread that wrote to it. All these optimizations are described more fully in Leffler [2003b].


Exercises

13.1 Is TCP a transport-, network-, or link-layer protocol?

13.2 How does IPv4 identify the next-higher-level protocol that should process an incoming message? How might this dispatching differ in other networking architectures?

13.3 How many hosts can exist in an IPv4 subnet with a mask of 255.255.255.0?

13.4 What is a broadcast message? How are broadcast messages identified in IPv4? How are IPv6 broadcast messages identified?

13.5 Why are TCP and UDP protocol control blocks kept on separate lists?

13.6 Why does the output routine, rather than the socket-layer send routine (sosend()), check the destination address of an outgoing packet to see whether the destination address is a broadcast address?

13.7 Why does FreeBSD not forward broadcast messages?

13.8 Why does the TCP header include a header-length field even though it is always encapsulated in an IP packet that contains the length of the TCP message?

13.9 What is the flow-control mechanism used by TCP to limit the rate at which data are transmitted?

13.10 How does TCP recognize messages from a host that are directed to a connection that existed previously but that has since been shut down (such as after a machine is rebooted)?

13.11 When is the size of the TCP receive window for a connection not equal to the amount of space available in the associated socket's receive buffer? Why are these values not equal at that time?

13.12 What are keepalive messages? For what does TCP use them? Why are keepalive messages implemented in the kernel rather than, say, in each application that wants this facility?

13.13 Why is calculating a smoothed round-trip time important, rather than, for example, just averaging calculated round-trip times?

13.14 Why does TCP delay acknowledgments for received data? What is the maximum time that TCP will delay an acknowledgment?

13.15 Explain what the silly-window syndrome is. Give an example in which its avoidance is important to good protocol performance. Explain how the FreeBSD TCP avoids this problem.

13.16 What is meant by small-packet avoidance? Why is small-packet avoidance bad for clients (e.g., the X Window System) that exhibit one-way data flow and that require low latency for good interactive performance?

13.17 Describe three ways in which IPv6 differs from IPv4.

13.18 In IPv6, what protocol replaces ARP for translating IP addresses to hardware addresses?

13.19 What does the networking code use the network mask, or prefix, of a link to determine?

13.20 Why are there separate protocols for authentication and encryption in IPSec?

13.21 Why is the cryptographic subsystem implemented using two queues and two kernel threads?

13.22 What limitation of ARP does neighbor discovery overcome? How does it overcome this limitation?

13.23 How is the protection offered to packets by IPSec different in tunnel mode and transport mode?

*13.24 Why is the initial sequence number for a TCP connection selected at random, rather than being, say, always set to zero?

*13.25 In the TCP protocol, why do the SYN and FIN flags occupy space in the sequence-number space?

*13.26 Describe a typical TCP packet exchange during connection setup. Assume that an active client initiated the connection to a passive server. How would this scenario change if the server tried simultaneously to initiate a connection to the client?

*13.27 Sketch the TCP state transitions that would take place if a server process accepted a connection and then immediately closed that connection before receiving any data. How would this scenario be altered if FreeBSD TCP supported a mechanism where a server could refuse a connection request before the system completed the connection?

*13.28 Why does UDP match the completely specified destination addresses of incoming messages to sockets with incomplete local and remote destination addresses?

*13.29 Why might a sender set the Don't Fragment flag in the header of an IP packet?

*13.30 The maximum segment lifetime (MSL) is the maximum time that a message may exist in a network—that is, the maximum time that a message may be in transit on some hardware medium or queued in a gateway. What does TCP do to ensure that TCP messages have a limited MSL? What does IP do to enforce a limited MSL? See Fletcher & Watson [1978] for another approach to this issue.

*13.31 Why does TCP use the timestamp option in addition to the sequence number in detecting old duplicate packets? Under what circumstances is this detection most desirable?

**13.32 Describe a protocol for calculating a bound on the maximum segment lifetime of messages in an Internet environment. How might TCP use a bound on the MSL (see Exercise 13.30) for a message to minimize the overhead associated with shutting down a TCP connection?

**13.33 Describe path MTU discovery. Can FreeBSD take advantage of the fact that the MTU of a path has suddenly increased? Why or why not?

**13.34 What are the tradeoffs between frequent and infrequent transmission of router advertisements in IPv6?

**13.35 Since IPSec may call routines in the network stack recursively, what requirement does this place on the code?

**13.36 Describe three paths that a packet can take through the networking code. How and where is each path chosen?


References

Bellovin, 1996. S. Bellovin, "Defending Against Sequence Number Attacks," RFC 1948, available from http://www.faqs.org/rfcs/rfcl948.html, May 1996.

Cain et al., 2002. B. Cain, S. Deering, I. Kouvelas, B. Fenner, & A. Thyagarajan, "Internet Group Management Protocol, Version 3," RFC 3376, available from http://www.faqs.org/rfcs/rfc3376.html, October 2002.

Cerf & Kahn, 1974. V. Cerf & R. Kahn, "A Protocol for Packet Network Intercommunication," IEEE Transactions on Communications, vol. 22, no. 5, pp. 637-648, May 1974.

Clark, 1982. D. D. Clark, "Window and Acknowledgment Strategy in TCP," RFC 813, available from http://www.faqs.org/rfcs/rfc813.html, July 1982.

Conta & Deering, 1998. A. Conta & S. Deering, "Internet Control Message Protocol (ICMPv6) for the Internet Protocol Version 6 (IPv6) Specification," RFC 2463, available from http://www.faqs.org/rfcs/rfc2463.html, December 1998.

DARPA, 1983. DARPA, "A History of the ARPANET: The First Decade," Technical Report, Bolt, Beranek, and Newman, Cambridge, MA, April 1983.

Deering, 1989. S. Deering, "Host Extensions for IP Multicasting," RFC 1112, available from http://www.faqs.org/rfcs/rfc1112.html, August 1989.

Deering & Hinden, 1998a. S. Deering & R. Hinden, "Internet Protocol, Version 6 (IPv6)," RFC 2460, available from http://www.faqs.org/rfcs/rfc2460.html, December 1998.

Deering & Hinden, 1998b. S. Deering & R. Hinden, "IP Version 6 Addressing Architecture," RFC 2373, available from http://www.faqs.org/rfcs/rfc2373.html, July 1998.

Fletcher & Watson, 1978. J. Fletcher & R. Watson, "Mechanisms for a Reliable Timer-Based Protocol," in Computer Networks 2, pp. 271-290, North-Holland, Amsterdam, The Netherlands, 1978.

Fuller et al., 1993. V Fuller, T. Li, J. Yu, & K. Varadhan, "Classless Inter-Domain Routing (CIDR): An Address Assignment and Aggregation Strategy," RFC 1519, available from http://www.faqs.org/rfcs/rfc1519.html, September 1993.

Gilligan et al., 1999. G. Gilligan, S. Thomson, J. Bound, & W. Stevens, "Basic Socket Interface Extensions for IPv6," RFC 2553, available from http://www.faqs.org/rfcs/rfc2553.html, March 1999.

Gross & Almquist, 1992. P. Gross & P. Almquist, "IESG Deliberations on Routing and Addressing," RFC 1380, available from http://www.faqs.org/rfcs/rfc1380.html, November 1992.

Harkins & Carrel, 1998. D. Harkins & D. Carrel, "The Internet Key Exchange (IKE)," RFC 2409, available from http://www.faqs.org/rfcs/rfc2409.html, November 1998.

ISO, 1984. ISO, "Open Systems Interconnection: Basic Reference Model," ISO 7498, International Organization for Standardization, available from the American National Standards Institute, 1430 Broadway, New York, NY 10018, 1984.

Jacobson, 1988. V Jacobson, "Congestion Avoidance and Control," Proceedings of the ACM SIGCOMM Conference, pp. 314-329, August 1988.

Jacobson et al., 1992. V Jacobson, R. Braden, & D. Borman, "TCP Extensions for High Performance," RFC 1323, available from http://www.faqs.org/rfcs/rfc1323.htm1, May 1992.

KAME, 2003. KAME, "Overview of KAME Project," Web site, http://www.kame.net/project-overview.html#overview, December 2003.

Kent & Atkinson, 1998a. S. Kent & R. Atkinson, "Security Architecture for the Internet Protocol," RFC 2401, available from http://www.faqs.org/rfcs/rfc2401.htm1, November 1998.

Kent & Atkinson, 1998b. S. Kent & R. Atkinson, "IP Authentication Header," RFC 2402, available from http://www.faqs.org/rfcs/rfc2402.htm1, November 1998.

Kent & Atkinson, 1998c. S. Kent & R. Atkinson, "IP Encapsulating Security Payload (ESP)," RFC 2406, available from http://www.faqs.org/rfcs/rfc2406.html, November 1998.

Leffler, 2003a. S. Leffler, "Fast IPSec: A High-Performance IPSec Implementation," Proceedings of BSDCon 2003, September 2003.

Leffler, 2003b. S. Leffler, "Cryptographic Device Support for FreeBSD," Proceedings of BSDCon 2003, September 2003.

McDonald et al., 1998. D. McDonald, C. Metz, & B. Phan, "PF_KEY Key Management API, Version 2," RFC 2367, available from http://www.faqs.org/rfcs/rfc2367.html, July 1998.

McQuillan & Walden, 1977. J. M. McQuillan & D. C. Walden, "The ARPA Network Design Decisions," Computer Networks, vol. 1, no. 5, pp. 243–289, 1977.

Mogul, 1984. J. Mogul, "Broadcasting Internet Datagrams," RFC 919, available from http://www.faqs.org/rfcs/rfc9l9.html, October 1984.

Mogul & Deering, 1990. J. Mogul & S. Deering, "Path MTU Discovery," RFC 1191, available from http://www.faqs.org/rfcs/rfc1191.html, November 1990.

Mogul & Postel, 1985. J. Mogul & J. Postel, "Internet Standard Subnetting Procedure," RFC 950, available from http://www.faqs.org/rfcs/rfc950.html, August 1985.

Nagle, 1984. J. Nagle, "Congestion Control in IP/TCP Internetworks," RFC 896, available from http://www.faqs.org/rfcs/rfc896.html, January 1984.

Narten et al., 1998. T. Narten, E. Nordmark, & W. Simpson, "Neighbor Discovery for IP Version 6 (IPv6)," RFC 2461, available from http://www.faqs.org/rfcs/rfc2461.html, December 1998.

Nesser, 1996. P. Nesser, "An Appeal to the Internet Community to Return Unused IP Networks (Prefixes) to the IANA," RFC 1917, available from http://www.fags.org/rfcs/rfc1917.html, February 1996.

Postel, 1980. J. Postel, "User Datagram Protocol," RFC 768, available from http://www.faqs.org/rfcs/rfc768.html, August 1980.

Postel, 1981a. J. Postel, "Internet Protocol," RFC 791, available from http://www.faqs.org/rfcs/rfc791.html, September 1981.

Postel, 1981b. J. Postel, "Transmission Control Protocol," RFC 793, available from http://www.faqs.org/rfcs/rfc793.html, September 1981.

Postel, 1981c. J. Postel, "Internet Control Message Protocol," RFC 792, available from http://www.faqs.org/rfcs/rfc792.html, September 1981.

Postel et al., 1981. J. Postel, C. Sunshine, & D. Cohen, "The ARPA Internet Protocol," Computer Networks, vol. 5, no. 4, pp. 261–271, July 1981.

Sakane, 2001. S. Sakane, Simple Configuration Sample of IPsec/Racoon, available at http://www.kame.net/newsletter/20001119, September 2001.

Scheifler & Gettys, 1986. R. W. Scheifler & J. Gettys, "The X Window System," ACM Transactions on Graphics, vol. 5, no. 2, pp. 79–109, April 1986.

Simpson, 1995. W. Simpson, "IP in IP Tunneling," RFC 1853, available from http://www.faqs.org/rfcs/rfc1853.html, October 1995.

Thomson & Huitema, 1995. S. Thomson & C. Huitema, "DNS Extensions to Support IP Version 6," RFC 1886, available from http://www.faqs.org/rfcs/rfc1886.html, December 1995.


Part V: System Operation

    Chapter 14. Startup and Shutdown


Chapter 14. Startup and Shutdown

When a computer is powered on, there is nothing running on the CPU. For a program to be set running, the binary image of the program must first be loaded into memory from a storage device. Many systems automatically start programs that reside in nonvolatile storage devices such as read-only memories (ROMs). Once the image of the program is loaded, the CPU must be directed to start execution at the first memory location of the loaded program. This process of bootstrapping a program into execution starts a program running on a CPU.

In this chapter, we examine how the FreeBSD kernel, or any other similar program, is bootstrapped. We then study the operation of the system during the initialization phase, which takes the system from a cold start to the point at which user-mode programs can be run. A final section examines topics that are related to the startup procedure. These topics include configuring the kernel load image, shutting down a running system, and debugging system failures.


14.1. Overview

The FreeBSD kernel is only a program, albeit a complex one. Like any program, its binary image resides in a file on a filesystem until it is loaded and set running. FreeBSD presumes that the executable image of the kernel resides in a file named /boot/kernel/kernel on a filesystem that is designated as the root filesystem. The initial bootstrap mechanism is machine dependent. Often a small bootstrap program is placed in a reserved area near the start of the primary disk. This program is usually limited to a small area—as little as one 512-byte disk sector—and simply loads a larger program from the following area of the disk. This program, or some other mechanism, is usually used to load and run a special program, named boot. The boot program's task is to load and initialize the executable image of a program and to start that program running. Boot may come from the same storage device as the file that it bootstraps, or it may be loaded from a network file server.

When the boot program is started, it loads /boot/kernel/kernel and then begins a countdown that can be interrupted by the user. If the boot program's countdown is interrupted, it provides the user with a command line interpreter on the console and waits for input. The list of commands is shown in Table 14.1. The command line interpreter helps debug problems occurring at boot time. The most commonly used commands are the unload command, to remove the default kernel, followed by the load command, to load an alternative kernel. Typically the alternative kernel being loaded is /boot/kernel/kernel.old, saved by the kernel build system when installing a new kernel in case the new kernel proves to be faulty.

Table 14.1. Boot Program Commands.

Command

Description

beachestat

get disk block cache statistics

boot

boot a file or loaded kernel

autoboot

boot automatically after a delay

help

detailed help

?

list available commands

show

show variable(s)

set

set a variable

unset

unset a variable

echo

echo back some input, analogous to echo in the shell

read

emulate the shell command read

more

show contents of a file with paged output

lsdev

list all devices

include

read commands from a file

1s

list files that the boot program can see

load

load a kernel or module

unload

unload all modules

lsmod

list all loaded modules

pnpscan

scan for plug-and-play (PnP) devices

pnpload

load modules for plug-and-play devices

The kernel is started by issuing the boot command, either by the user or because the countdown has reached 0. The boot command starts by initializing the CPU, ensuring that the virtual-address translation is turned off and that hardware interrupts are disabled. The loaded program is responsible for enabling these facilities and any additional hardware, such as I/O devices, that it intends to use.

When the FreeBSD kernel is loaded by the boot program, the kernel goes through several stages of hardware and software initialization in preparation for normal system operation. The first stage is responsible for setting up the initial state of the CPU, including the run-time stack and virtual-memory mapping. Memory mapping, including virtual-address translation, is enabled early in the startup procedure to minimize the amount of special-purpose assembly-language code that those porting the kernel must write. Once virtual-memory mapping is enabled, the system does machine-dependent initializations and then machine-independent initializations. The machine-dependent operations include setting up mutexes and virtual-memory page tables, and configuring I/O devices; the machine-independent actions include mounting the root filesystem and initializing the myriad system data structures. This order is necessary because many of the machine-independent initializations depend on the I/O devices being initialized properly.

Following the setup of the machine-independent portions of the kernel, the system is in operational status. System processes are created and made runnable, and user-level programs are brought in from the filesystems to execute. At this point, the system is ready to run normal applications.


14.2. Bootstrapping

Bootstrapping a program is a machine-dependent operation. On most machines, this operation is supported by a Basic Input/Output System (BIOS). The BIOS is resident in nonvolatile storage and is invoked automatically when the CPU is reset. The facilities provided by the BIOS are expected to support the bootstrap of standalone programs. Most BIOSs also execute diagnostic operations when a machine is reset to ensure that the hardware is functioning properly.

The boot Program

The BIOS does not understand the format of the FreeBSD filesystem. Instead, the startup procedure reads a program from a reserved area of the boot disk. This procedure ultimately results in the execution of the FreeBSD boot program. This program is a general-purpose standalone program that the system can use to load and execute other standalone programs. A standalone program is a program that is capable of operating without the assistance of the FreeBSD kernel. Standalone programs usually are linked with the standalone I/O library, a library that supports a FreeBSD-like I/O interface on a variety of hardware devices. The standalone I/O library provides these facilities through a collection of standalone device drivers and a library of routines that support reading of files from FreeBSD filesystems that reside on the devices. The boot program is stored in a location accessible to the BIOS, which is the first few sectors of a system disk.

Once the boot program has been loaded and started, it must load the file containing the executable image of the program to be bootstrapped and then must start the loaded program running. To load the appropriate file, boot must know the pathname of the file to be loaded and the hardware device on which the file resides. The boot program usually has a default device and program name from which it tries to load. Often, this default is stored in the boot sectors of the boot disk along with the boot program. The BIOS communicates the bootstrapping information to the boot program by initializing the run-time stack and then placing the parameters on the stack in the same way that the FreeBSD kernel passes arguments to programs. Alternatively, a user may type in the device and program name to be used.

Boot always loads programs at a well-known memory location. Since boot is initially loaded in memory at the same well-known location, it must copy its own image to another place in memory to avoid loading on top of itself the image of the program that it bootstraps (see Figure 14.1). This relocation implies that the boot program must be created with its starting address set to the memory location at which it will be copied; otherwise, references to data structures in the boot program will access the wrong memory locations after boot is copied (remember that boot operates with virtual-address translation disabled).

Figure 14.1. Placement of the boot program in memory.


14.3. Kernel Initialization

When the FreeBSD kernel is started by the boot program, it does an initialization in preparation for the execution of application programs. The initialization process is divided into roughly three stages. The first stage requires handcrafted assembly-language code to set up the hardware for the more generic non-assembly-language code to operate. The second stage loads and initializes kernel modules that implement all the kernel's internal services, including configuring and initializing the I/O devices on the machine. The third stage starts up the system-resident processes that compose the foundation for the normal FreeBSD run-time environment and then executes the user-level-startup scripts.

Assembly-Language Startup

The first steps taken by the system during initialization are carried out by assembly-language code. This work is highly machine dependent and includes the following:

  • Setting up the run-time stack

  • Identifying the type of CPU on which the system is executing

  • Calculating the amount of physical memory on the machine

  • Enabling the virtual-address-translation hardware

  • Initializing the memory-management hardware

  • Setting up tables for SMP operation, if necessary

  • Crafting the hardware context for process 0

  • Invoking the initial C-based entry point of the system

Although the details of these steps vary from architecture to architecture, the broad outline described here is applicable to any machine on which FreeBSD runs.

When the boot program starts the FreeBSD kernel running, it sets up only two components of the machine state:

  1. All interrupts are blocked.

  2. The hardware address-translation facility is disabled so that all memory references are to physical memory locations.

The boot program also passes to the kernel the identity of the boot device and a set of boot flags. The FreeBSD kernel presumes nothing else about the state of the machine on which it is running.

The kernel is loaded into physical memory at a known location—often, at the lowest physical address. In normal operation, the address-translation hardware is enabled, and the kernel image is mapped into virtual memory starting at an address near the top of the address space. Before the address translation has been enabled, the assembly-language startup code must convert all absolute addresses from their virtual-memory location to their physical-memory location. The kernel is usually loaded into contiguous physical memory, so the translation is simply a constant offset that can be saved in an index register.

A second task of the startup code is to identify the type of CPU on which the system is executing. Often, older versions of the CPU support only a subset of the complete instruction set. For these machines, the kernel must emulate the missing hardware instructions in software. For most architectures, FreeBSD can be configured such that a single kernel load image can support all the models in an architecture family. The startup code may also call machine-dependent code to initialize the CPU or virtual-memory subsystem.


14.4. Kernel Module Initialization

In earlier versions of BSD the initialization of the underlying hardware was handled exclusively by custom-written code. An intimate understanding of the entire operating system was needed for someone to add a new service or subsystem that slowed the evolution of the kernel. In FreeBSD, a method has been devised to break down all kernel services into modules and to logically sequence these modules at boot time, making it easier to experimentally add new features to the operating system. All subsystems are listed in the /sys/sys/kernel.h include file.

There are two types of kernel modules used in FreeBSD. Modules that can be loaded and unloaded from the system at run time are called kernel loadable modules and are discussed later in this section. Kernel modules that must be loaded at boot time and cannot be removed once they are running are considered to be permanent kernel modules. A permanent module is exported to the system using the SYSINIT macro to identify the module, its initialization routine, and the order in which it should be called.

SYSINIT(name, subsystem, order, function, identifier)

All modules are organized into a two-level hierarchy to provide an orderly startup of the system. The subsystem argument is the first level of the hierarchy. Each subsystem is assigned a particular numeric constant that creates the first-level ordering of the modules to be loaded. The second level of the hierarchy is handled by the order argument. If two modules are in the same subsystem, the order shows which comes first. The function argument is the initialization function that the kernel calls at system startup.

After the assembly-language code has completed its work, it calls the first kernel routine written in C: the mi_startup() routine. The mi_startup() routine first sorts the list of modules that need to be started and then calls the function routine of each. Some initializations are machine dependent. They relate only to a particular hardware architecture. Other initializations are machine independent. They include services that do not need to be aware of the underlying hardware. Each kernel module is implemented either as being hardware dependent or independent according to its needs. Once the assembly-language startup sequence has completed, any other assembly-language code necessary to bring up the system is called through C entry points. This separation of dependent and independent initializations makes the task of coding the startup sequence on a new platform easier than in earlier releases of BSD.

Basic Services

Before FreeBSD can do any useful work, it must set up some basic services that are used by all the subsystems inside the kernel. These services include support for mutexes, the lock manager, and the kernel memory manager. They are shown in Table 14.2. All these services are initialized early in the startup sequence so that they can be used by the rest of the kernel.

Table 14.2. Basic services.

Module

First Routine

SI_SUB_MTX_POOL_STATIC

mtx_pool_setup_static()

SI_SUB_LOCKMGR

lockmgr_init()

SI_SUB_VM

vm_mem_init()

SI_SUB_KMEM

kmeminit()

SI_SUB_KVM_RSRC

vmmapentry_rsrc_init()

SI_SUB_WITNESS

witness _initialize()

SI_SUB_MTX_POOL_DYNAMIC

mtx_pool_setup_dynamic()

SI_SUB_LOCK

selectinit()

SI_SUB_EVENTHANDLER

eventhandler_init()

SI_SUB_KLD

linker _init()

SI_SUB_CPU

cpu_startup()

Mutexes, which were covered in Section 4.3, are initialized in two different pools, one static and one dynamic. The initialization is split because dynamic pools use the kernel memory allocator that must be setup after the static mutexes. Because of the tight dependency of kernel services on each other, changes to the ordering of kernel services need to be carefully considered.

Directly after the lock manager is initialized the system enables the virtual memory system with a call to vm_mem_init(). Once the vm_mem_init() routine has completed its work, all memory allocations by the kernel or processes are for virtual addresses that are translated by the memory management hardware into physical addresses. With the virtual memory system running, the kernel now starts its own internal allocator. One last piece of memory subsystem initialization is to set limits on the resources used by the kernel virtual memory system, which is handled by the vmmapentry_rsrc_init() routine. Kernel modules can now ask for memory using the kernel's malloc() routine.

Moving from a uniprocessor kernel to one that supports SMP required the locking of the kernel's data structures against multiple, independent threads. Debugging multiprocessor code is difficult enough at the process level where the operating system can provide some help to the programmer. In an operating-system kernel this problem becomes more complex because a crashed kernel is hard to debug. To aid kernel debugging, FreeBSD added a kernel library that can watch all the locks being taken and released. The witness_initialize() routine instantiates the SI_SUB_WITNESS module as a basic service to provide the witness library to the modules that handle locking.

Some services require their locks to be allocated as part of the basic services. The select system call and the subsystem that handles kernel objects are two such services. The SI_SUB_LOCK module provides a place for services to register their need to have their lock initialization routines called early in the startup sequence.

The event-handler module allows various services to register functions to be called by the kernel when an event occurs. The event-handler service is heavily used by the shutdown sequence, described in Section 14.6, but also handles such diverse events as a device being cloned; the virtual memory system running low on memory; and processes being forked, executed, and exiting. The event-handler module, SI_SUB_EVENTHANDLER is started as part of the basic services so that other modules can use it when they are started.

The last of the basic services to be started is the kernel-module loader that loads dynamic kernel modules into the system at boot or run-time. The module_init() routine creates the basic data structures to handle module loading and unloading as well as registering an event handler to unload all the modules when the system is shut down. The kernel-module loader's use of event handlers is why this module must be initialized after the event-handler module just discussed.

With the basic services running, the kernel can now finish bringing up the CPU. The SI_SUB_CPU_module is really divided among several submodules that are responsible for different components of system startup. The first submodule to be initialized is the Advanced Programmable Interrupt Controller (APIC) that provides hardware interrupt support as well as coordination between multiple CPUs on SMP systems. The apic_init() routine starts the APIC device and then probes the system for other CPUs. Without the APIC device, multiple CPUs in an SMP system would have no way to coordinate their work, so this step is taken early in the system startup sequence. Although the system is not yet ready to start the other CPUs, it does allocate and initialize the data structures that support them. The next subsystem to start is the CPU itself. Although it is already running, having been started by the assembly-language boot code, several other pieces of initialization take place here. The real-time clock is started by the startrtclock() routine, and the CPU information is printed on the console, as are the memory statistics. The buffer system is also started by the CPU module so that the kernel can read data from disk. FreeBSD is heavily dependent on stable storage for swapping and paging memory and for retrieving kernel modules and programs. So the buffer system is set up early in the boot sequence to give the kernel early access to these resources.

On SMP systems, the cpu_mp module is forced to start after the APIC and CPU modules because its order argument is set to SI_ORDER_SECOND, whereas the APIC and CPU both have SI_ORDER_FIRST. Because the APIC is absolutely necessary to an SMP system, it is necessary to put the SMP initialization after the APIC initialization. The cpu_mp's initialization function, mp_start(), calls machine-dependent routines that do the real work of starting the other processors. When this routine completes, all the processors in an SMP system are active but will not be given any work until after the entire startup sequence completes.

Kernel Thread Initialization

The kernel has several processes that are set up at boot time and that are necessary whenever the kernel is running. The modules that represent the kernel processes are shown in Table 14.3. The swapper, init, and idle processes are all set up after the CPU has been initialized, but they are not scheduled to run. No processes or kernel threads execute until all the kernel modules have been initialized.

Table 14.3. Kernel process modules.

Module

First Routine

SI_SUB_INTRINSIC

proc0_init()

SI_SUB_VM_CONF

vm_init_limits()

SI_SUB_RUN_QUEUE

runq_init()

SI_SUB_KTRACE

ktrace_init()

SI_SUB_CREATE_INIT

create_init()

SI_SUB_SCHED_IDLE

idle_setup()

Once the kernel is running, all new processes are created by forking an existing process. However, as there are no processes when the system is first started, the startup code must handcraft the first process. The swapper is always the first process and gets PID 0. Creation of the swapper process started in the assembly-language startup, but now the system has enough services working to almost finish the job. The proc0_init() routine not only sets up process 0, but also initializes all the global data structures for handling processes and threads, the file descriptor table, and the limits structures. The proc0_init() routine also creates a prototype virtual-memory map that will be the prototype for all the other processes that will eventually be created.

After the data structures for the swapper process have been initialized, the system then creates the init process by calling the create_init() routine. The init process is set up directly after the swapper process so that it will always have PID 1. Because privileged users communicate with the init process by sending it signals, ensuring that it has a well-known PID means that users will not have to look it up before being able to communicate.

Each CPU in a system has an idle process. It is the idle process's responsibility to halt a CPU when there is no work for it to do. Once the swapper and init processes have been set up, the kernel initializes an idle process for each CPU in the system. If the system has only one CPU, then only one idle process is created. Like all the other kernel processes, the idle process is not started at this point but only created.

Device Module Initialization

With all the kernel's basic services in place, and the basic processes created, it is now possible to initialize the rest of the devices in the system that includes the disks, network interfaces, and clocks. Table 14.4 (on page 602) shows the main components used to initialize the device modules.

Table 14.4. Device modules.

Module

First Routine

SI_SUB_MBUF

mbuf_init()

SI_SUB_INTR

intr_init()

SI_SUB_SOFTINTR

start_softintr(), start_netisr()

SI_SUB_DEVFS

devfs_init(), devs_set_ready()

SI_SUB_INIT_IF

if_init()

SI_SUB_DRIVERS

many different routines

SI_SUB_CONFIGURE

configure_first()

SI_SUB_VFS

vfsinit()

SI_SUB_CLOCKS

initclocks()

SI_SUB_CLIST

clist_init()

Before any devices can be initialized—in particular, network interfaces—the mbuf subsystem must be set up so that the network interfaces have a set of buffers they can use in their own initialization. The mbuf subsystem initialization is handled by the SI_SUB_MBUF module and its mbuf_init() routine. As we discussed in Section 11.3, the mbuf subsystem has two kinds of memory to manage: small mbufs and mbuf clusters. Each type of memory is allocated from its own kernel memory area to keep the allocator simple and so that the remaining memory does not get fragmented by different-size allocations.

At this point in the startup sequence, hardware interrupts are not enabled on the system. The kernel now sets up all the interrupt threads that will handle interrupts when the system begins to run. The interrupts are set up by two modules: SI_SUB_INTR, which sets up the interrupt threads that handle device interrupts, and SI_SUB_SOFTINTR, which creates soft-interrupt threads. Soft-interrupt threads are used by services that handle asynchronous events that are not generated by hardware. Soft-interrupt threads provide the soft clock, which supports the callout subsystem. They also provide the network thread that dequeues inbound packets from the network interface queues and moves them through the network stack.

As part of bringing up real hardware devices, the kernel first initializes the device filesystem and then readies the network stack to handle devices with a call to the if_init() routine. The if_init() routine does not initialize any network interfaces; it only sets up the data structures that will support them. Finally, the devices themselves are initialized by the SI_SUB_DRIVERS and SI_SUB_CONFIGURE modules. All devices in the system are initialized by autoconfiguration as described in Section 7.5.

Once the devices are configured, the virtual filesystem is initialized. Bringing up the VFS is a multistage process that entails initializing the VFS itself, the vnode subsystem, and the name cache and pathname translation subsystem that maps pathnames to inodes. Support for named pipes is also initialized as part of the VFS.

The next systems to be set up are those relating to the real-time clock provided by the hardware. The initclocks() routine, which is a part of the SI_SUB_CLOCKS module, calls the architecture's specific cpu_initclocks() routine to initialize the hardware clock on the system and start it running. Once the hardware clock is running other services such as support for the Network Time Protocol (NTP), device polling, and the time counter are started. The last data structures to be initialized are in the terminal subsystem by the SI_SUB_CLIST module. All that needs to be done is to allocate the initial set of cblocks and add them to the clist.

Kernel Loadable Modules

Some kernel modules can be loaded, shut down, and unloaded while the system is running. Providing a system where kernel services can be loaded and unloaded at run time has several advantages over a system where all kernel services must be linked in at build time. For systems programmers, being able to load and unload modules at run time means that they are able to develop their code more quickly. Only those modules absolutely necessary to the system, such as the memory manager and scheduler, need to be directly linked into the kernel. A kernel module can be compiled, loaded, debugged, unloaded, modified, compiled, and loaded again without having to link it directly into the kernel or having to reboot the system. When a system is placed in the field, the use of kernel modules makes it possible to upgrade only selected parts of the system, as necessary. Upgrading in the field is absolutely necessary in embedded applications where the system may be physically unreachable, but it is also convenient in more traditional environments where many systems might have to change at the same time—for instance, in a large server farm.

One problem with allowing kernel modules to be loaded and unloaded at run time is that of security. In versions of BSD before FreeBSD, the kernel was inviolate and completely protected from user change at run time. The only way to interact with a running kernel was through the syscall interface. System calls were defined at kernel build time and only provided a narrow channel of communication. This tightly controlled interface provided a layer of security. Although users could crash their own programs, absent a serious bug in the kernel, they could not crash the operating system or other users' processes. With the advent of loadable kernel modules, any user that can acquire root privileges can modify the underlying kernel. Certain services cannot be unloaded, which is a form of protection, but any service that is properly written can be loaded, including those that are malicious. Once a malicious module is loaded into the kernel, there is no protection against it wreaking havoc. A serious flaw in the kernel-module system is that it does not support the digital signing of modules. If a developer wishes to provide a service via a loadable kernel module to the general public—for instance, as a part of a larger application—it would be helpful to consumers of the module if the kernel could verify that the module really came from the stated developer. As yet, a module signing and verification service is not a part of the loadable kernel-module system. For these security reasons, most people that run FreeBSD in a commercial setting continue to use wholly contained kernels and do not load random services at run time.

Loadable kernel modules are declared in the following way:

DECLARE_MODULE(name, data, subsystem, order)

Each module has a name, subsystem, and order that serve the same purposes here as they do in the SYSINIT macro. The key difference is the use of the data argument, which is a data structure that is defined in the following way:

int (*modeventhand_t)(
        struct module *module,
        int command,
        void *argument);

typedef moduledata = {
        const char *name;
        modeventhand_t event_handler;
        void *data;
} moduledata_t;

All modules have an associated version that is declared with the MODULE VERSION macro. Without a version it would be impossible to differentiate between different revisions of the same module, making field upgrades difficult. One last macro used by kernel modules is the SYSCALL_MODULE_HELPER that developers use to add new system calls to the kernel.

To have kernel modules that can be loaded both at boot time and at run time, two different cases must be handled. When a module is loaded at boot time, it has already been processed by the kernel's build process, which means that its system-call entry points and other data are already known to the kernel. This knowledge simplifies the loading process. All that needs to be done is to call the module's event handler with the MOD_LOAD command. At run time, the module needs to be loaded into memory, registered with the kernel, and its system calls dynamically added to the system-call table. Once all that work is done, it can be initialized by calling its event handler. All the run-time loading is handled by the kldload system call and the module_register() routine. To keep the interface that programmers use simple, all this functionality is hidden by the DECLARE_MODULE macro and the use of the single event-handler routine. When creating a kernel module, a programmer needs to be concerned only with writing the module event handler and exporting the module handler's system calls via the macros.

Interprocess Communication Startup

The modules involved in the startup of the IPC interfaces are shown in Table 14.5. The first three modules to be loaded and started as part of supporting IPC are System V semaphores, shared memory, and message queues. These local-IPC facilities were covered in Section 11.8, and their initialization is simple. What makes them interesting is that they are the first modules that can be loaded and unloaded at run time.

Table 14.5. Interprocess communication modules.

Module

First Routine

SI_SUB_SYSV_SEM

sysvsem_modload()

SI_SUB_SYSV_SHM

sysvshm_modload()

SI_SUB_SYSV_MSG

sysvmsg_modload()

SI_SUB_PROTO_IF

if_check()

SI_SUB_PROTO_DOMAIN

domaininit()

SI_SUB_PROTO_IFATTACHDOMAIN

if_attachdomain()

We will use the SI_SUB_SYSV_SEM module as an example. It implements System V semaphores. This module is declared to the system as follows:

static moduledata_t sysvsem_mod = {
    "sysvsem",
    &sysvsem_modload,
    NULL
};

SYSCALL_MODULE_HELPER(semsys);
SYSCALL_MODULE_HELPER(_semctl);
SYSCALL_MODULE_HELPER(semget);
SYSCALL_MODULE_HELPER(semop);

DECLARE_MODULE(sysvsem, sysvsem_mod,
    SI_SUB_SYSV_SEM, SI_ORDER_FIRST);
MODULE_VERSION(sysvsem, 1);

All the system calls provided by the module are declared in the SYSCALL_MODULE_HELPER macros and should be familiar from Section 11.8. The DECLARE_MODULE macro ensures that an entry in the startup sequence is added for the module and that its order is first. There are no other modules in this subsystem so the order argument is not important. The routine that the startup sequence calls first is not sysvsem_modload() but module_register_init(), which then calls the module's sysvsem_modload() routine. Since this module was linked into the kernel at build time, all its system calls and other data are already linked with the kernel. Once the module_register_init() routine completes, the module is loaded and ready for use.

Following the local-IPC subsystems, several pseudo-devices are started. The pseudo-devices represent diverse services within the kernel that present themselves as devices to the rest of the system. Examples of pseudo-devices include the frame buffer, virtual LAN support, network firewalling, and the cryptography subsystems. All the pseudo-devices are declared as kernel loadable modules.

The last three modules—SI_SUB_PROTO_IF, SI_SUB_PROTO_DOMAIN, and SI_SUB_PROTO_IFATTACHDOMAIN—initialize the code that supports the networking support discussed in Part IV. The SI_SUB_PROTO_IF module has a single routine, if_check(), that checks each network interface in the system to see if it has been initialized correctly. The SI_SUB_PROTO_DOMAIN is a set of modules that initialize the various communication domains supported by the kernel. Once the basic data structures for domains are initialized by the domaininit() routine, the communication domains themselves are initialized. Each domain is responsible for declaring itself to the system by using the DOMAIN_SET macro that provides an initialization routine for the kernel to call, just like the SYSINIT macro. Unlike many of the preceding services, which were declared as modules, network domains are declared such that they cannot be removed. Domains are locked into the kernel once they are loaded because the socket layer may be using a domain's data structures for active communication, and the result of unloading a domain while it is in use is undefined. Making it possible to unload a communication domain at run time will take many changes to the socket layer and domains modules.

Start Kernel Threads

The collection of modules shown in Table 14.6 finishes the setup of all the threads necessary to have a working kernel. The swapper, init, pagezero, pagedaemon, bufdaemon, vnlru, and syncer kernel threads, as well as the kernel threads for the network filesystem (NFS), are started here. To this point the system has been running in uniprocessor mode. On a multiprocessor system the SI_SUB_SMP module calls the last of the machine-dependent code, the release_aps() routine, to start up the other processors in the system. Now it is time to start the system running. The SI_SUB_RUN_SCHEDULER module is treated specially in the startup sequence because it must always execute last. In the list of modules, it is given the highest possible subsystem number, all 1s, so that it is not accidentally placed earlier in the list. The scheduler() routine is called and never returns. It immediately begins scheduling kernel threads and user-level processes to run on the system's CPUs.

Table 14.6. Kernel thread modules.

Module

First Routine

SI_SUB_INTRINSIC_POST

proc0_post()

SI_SUB_KTHREAD_INIT

kick_init()

SI_SUB_KTHREAD_PAGE

vm_pageout()

SI_SUB_KTHREAD_VM

vm_daemon()

SI_SUB_KTHREAD_BUF

buf_daemon()

SI_SUB_KTHREAD_UPDATE

vnlru_proc()

SI_SUB_KTHREAD_IDLE

ald_daemon()

SI_SUB_SMP

release_aps()

SI_SUB_RUN_SCHEDULER

scheduler()


14.5. User-Level Initialization

With the start of the init process, most of the system is operating and functional. There are several additional steps taken between this point and the time a user sees a prompt to sign on to the system. All these actions are done by user-level programs that use the standard FreeBSD system-call interface that has been described in previous chapters. We shall briefly examine the steps that take place in a typical system.

/sbin/init

The /sbin/init program is invoked as the final step in the bootstrapping procedure. The parameters specified at the time FreeBSD was bootstrapped are passed to init in a machine-dependent fashion. Init uses the values of these flags to determine whether it should bring up the system to single-user or to multiuser operation and whether it should check the consistency of its disks with the fsck program. In single-user operation, init forks a process that invokes the standard shell, /bin/sh. The standard input, output, and error descriptors of the process are directed to the system's console terminal, /dev/console. This shell then operates normally but with superuser privileges until it terminates.

In multiuser operation, init first spawns a shell to interpret the commands in the file /etc/rc, which is the root of a set of system startup scripts that do all the user-level initialization of the system. If the /etc/rc script completes successfully, init then forks a copy of itself for each terminal device that is marked for use in the file /etc/ttys. These copies of init invoke other system programs, such as /usr/libexec/getty, to manage the standard sign-on procedure. Process 1 always acts as the master coordinating process for system operation. It is responsible for spawning new processes as terminal sessions are terminated and for managing the shutdown of a running system.

System Startup Scripts

The /etc/rc file is mostly empty and simply serves to order and execute the various system startup scripts contained in the /etc/rc.d directory. Two files, /etc/rc.conf and /etc/defaults/rc.conf, control which user-level services are started at boot time. Each of these files is loaded by the system startup scripts when they execute. /etc/defaults/rc.conf contains the default values for various shell variables that control whether a service is to be started. Administrators override the defaults by placing different values for the same shell variables into /etc/rc.conf.

For example, to enable the use of the Secure Shell (ssh) at boot time, the following line would be placed into /etc/rc.conf:

sshd_enab1e ="yes"

The heart of the rc script system is a program called reorder that takes a set of shell scripts as input, works out their interdependencies, and then outputs an ordered list of names. Each startup script declares the modules that it requires as well as those it provides. The reorder programs uses these REQUIRE and PROVIDE statements to determine the proper order in which to run the startup scripts.

One housekeeping task is to check the local filesystems after a system crash. If the system is not booted with the fastboot option, then the /etc/rc.d/fsck script carries out filesystem checks. In versions of BSD before FreeBSD, the filesystem checks were absolutely necessary and had to be carried out before any other work, but with the advent of soft updates (Section 8.6) and other filesystem changes, that is no longer necessary. The program /sbin/fsck checks filesystem consistency and repairs damaged filesystems. Normally, fsck is invoked from the /etc/rc.d/fsck script to examine and repair each filesystem before the latter is mounted. When the system is initially booted, the root filesystem is mounted read-only. If the root filesystem requires repairs, FreeBSD does a variant of the mount system call that requests the kernel to reload all its root-filesystem data structures. Reloading ensures consistency between the data in the kernel memory and any data in the filesystem that were modified by fsck. Having the root filesystem mounted readonly ensures that the kernel will not have any modified data in memory that cannot be reloaded.

Following the filesystem checks, the filesystems are mounted, the root filesystem is updated to be writable, and any devices that are to be used for swapping and paging are enabled. Disk quotas are then checked and enabled, and the system starts the background processes that implement various system services. These processes include /usr/sbin/cron, the program that executes commands at periodic intervals; /usr/sbin/accton, the program that enables system accounting; and /usr/sbin/syslogd, the system error-logging process. Each of these processes is started from its own startup script in /etc/rc.d.

/usr/libexec/getty

The /usr/libexec/getty program is spawned by init for each hardware terminal line on a system. This program is responsible for opening and initializing the terminal line. As a precaution against another process opening the line and snooping on the user's input, getty uses the revoke system call to revoke access to any open descriptors on the line (see Section 6.6). It then creates a new session for the line and requests that the terminal be made the controlling terminal for the session. The getty program sets the initial parameters for a terminal line and establishes the type of terminal attached to the line. For lines attached to a modem, getty can be directed to accept connections at a variety of baud rates. Getty selects this baud rate by changing the speed of the terminal line in response to a break character or a framing error, typically generated as a result of the user hitting a break key. A user can hit successive break keys to cycle through several line speeds until the proper one is found. Getty's actions are driven by a terminal-configuration database that is located in the file /etc/gettytab.

Getty finally reads a login name and invokes the /usr/bin/login program to complete a login sequence.

/usr/bin/login

The login program is responsible for signing a user onto the system; it is usually invoked by /usr/libexec/getty with the name of the user who wishes to log into the system. Login prompts the user for a password (after turning off terminal echoing if possible). If the password supplied by the user encrypts to the same value as that stored in the master password file /etc/master.passwd, login writes a record of the sign-on in various accounting files, initializes the user and group identifiers to those specified in the password and /etc/group files, and changes to the user's login directory. The user's login name is stored in the session structure using the setlogin system call so that it can be obtained reliably via the getlogin system call by programs that want to know the login name associated with a given process. Finally, login uses exec to overlay itself with the user's shell.

The login program is also invoked when a user enters the system through a network connection. Getty and init are bypassed for such connections; their functionality is subsumed by the daemon spawned when the network connection is established.


14.6. System Operation

In this section, we consider topics that are related to the system-startup procedure.

Kernel Configuration

The software that makes up a FreeBSD kernel is defined by a configuration file that is interpreted by the /usr/sbin/config program, which is invoked as part of the kernel build process. The kernel build process has become considerably more complex in FreeBSD and is now controlled by a set of Makefile targets. To build a kernel, the user invokes make in the following way:

make buildkernel KERNCONF=<kernel config file>

The buildkernel argument is a Makefile target that tells make to build a kernel but not to install it. KERNCONF is a Makefile variable that is set to the name of the kernel configuration. Once a kernel has been properly built, it is installed by running make in the following way:

make installkernel KERNCONF=<kernel config file>

One reason for this new build process is the need to build and install the necessary kernel modules, which were discussed in Section 14.4. The configuration file specifies the hardware and software components that should be supported by a kernel. The build process generates several output files, some of which are compiled and linked into the kernel's load image. It also creates a directory into which all the loadable kernel modules will be built. When the kernel is installed, its modules are installed as well. A complete description of the kernel build process is given in Hamby & Mock [2004].

System Shutdown and Autoreboot

FreeBSD provides several utility programs to halt or reboot a system, or to bring a system from multiuser to single-user operation. Safe halting and rebooting of a system require support from the kernel. This support is provided by the reboot system call.

The reboot system call is a privileged call. A parameter specifies how the system should be shut down and rebooted. This parameter is a superset of the flags passed by the boot program to the system when the latter is initially bootstrapped. A system can be brought to a halt (typically by its being forced to execute an infinite loop), or it can be rebooted to single-user or multiuser operation. There are additional controls that can be used to force a crash dump before rebooting (see the next subsection for information about crash dumps) and to disable the writing of data that are in the buffer cache to disk if the information in the buffer cache is wrong.

Automatic rebooting is also commonly done when a catastrophic failure is recognized. The system will reboot itself automatically if it recognizes an unrecoverable failure during normal operation. Failures of this type, termed panics, are all handled by the panic() subroutine.

When the system is shutting down, it goes through three separate phases:

  • Shutdown of all services that depend on the filesystem to store data

  • Shutdown of the filesystem itself

  • Shutdown of services that do not depend on the filesystem

These three phases are necessary because some services will want to write some final data to the filesystem before it is turned off and may not be able to restart cleanly if they cannot do so.

Services register event handlers with the kernel to provide an orderly shutdown of the system. Each event handler is declared with the following macro:

EVENTHANDLER_REGISTER(name, function, argument, priority)

The name shows in which part of the shutdown sequence the event handler's function will be called, as shown in Table 14.7. The argument allows the module to pass itself any private data necessary to turn itself off. The priority orders the shutdown routines within a phase. The priority serves the same purpose here as the order argument does in the SYSINIT macro in creating an orderly startup sequence. The priority is necessary to make sure that services do not go away while other services depend on them.

Table 14.7. Shutdown phases.

Name

Shutdown Phase

shutdown_pre_sync

Before the disks are synced.

shutdown_post_sync

After the disks are synced.

shutdown_final

Just before stopping the CPU.

The kernel shutdown routine first walks the list of shutdown_pre_sync functions and calls each in turn, and then it shuts down the filesystems on the local disks. With the filesystems in a quiescent state, it invokes all the shutdown_post_sync functions. A kernel core dump is made if requested—for example, if it was called because of a kernel panic. Kernel core dumps are written directly to the swap partition and not to a normal filesystem, which is why this step can come after the filesystems have been shut down. Finally, the kernel shutdown routine invokes all the functions registered in the shutdown_final group. It then goes into an infinite loop, awaiting a reset by the user.

System Debugging

FreeBSD provides several facilities for debugging system failures. The most commonly used facility is the crash dump: a copy of memory that is saved on secondary storage by the kernel when a catastrophic failure occurs. Crash dumps are created by the doadump() routine. They occur if a reboot system call is made in which the RB_DUMP flag is specified or if the system encounters an unrecoverable—and unexpected—error.

The doadump() saves the current context with a call to the savectx() routine and then invokes the dumpsys() routine to write the contents of physical memory to secondary storage. The precise location of a crash dump is configurable; most systems place the information at the end of the primary swap partition. This operation is done by the dump entry point of the configured disk driver.

A crash dump is retrieved from its location on disk by the /sbin/savecore program after the system is rebooted and the filesystems have been checked. It creates a file into which the crash-dump image is copied. Savecore also makes a copy of the initial kernel load image for use in debugging. The system administrator can examine crash dumps with the standard FreeBSD debugging program, gdb. The kernel is also set up so that a gdb debugger running on one machine can attach itself across a serial line to a kernel running on another machine. Once attached, it can set breakpoints, examine and modify kernel data structures, and invoke kernel routines on the machine being debugged. This form of source-level debugging is particularly useful in developing kernel device drivers, as long as the driver being developed is not the serial-line driver itself.

Passage of Information To and From the Kernel

In 4.3BSD and earlier systems, utilities that needed to get information from the kernel would open the special device /dev/kmem, which gave access to the kernel's memory. Using the name list from the kernel binary, the utility would seek to the address of the desired symbol and read the value at that location. Utilities with superuser privilege could also use this technique to modify kernel variables. Although this approach worked, it had four problems:

  1. Applications did not have a way to find the binary for the currently running kernel reliably. Using an incorrect binary would result in looking at the wrong location in /dev/kmem, resulting in wildly incorrect output. For programs that modified the kernel, using the wrong binary would usually result in crashing the system by trashing some unrelated data structure.

  2. Reading and interpreting the kernel name list is time-consuming. Thus, applications that had to read kernel data structures ran slowly.

  3. Applications given access to the kernel memory could read the entire kernel memory. Malicious programs could snoop the terminal or network input queues looking for users who were typing sensitive information such as passwords.

  4. As more of the kernel data structures became dynamically allocated, it became difficult to extract the desired information reliably. For example, in 4.3BSD, the process structures were all contained in a single statically allocated table that could be read in a single operation. In FreeBSD, process structures are allocated dynamically and are referenced through a linked list. Thus, they can be read out only one process entry at a time. Because a process entry is subdivided into many separate pieces, each of which resides in a different part of the kernel memory, every process entry takes several seeks and reads to extract through /dev/kmem.

To resolve these problems, 4.4BSD introduced the sysctl system call. This extensible-kernel interface allows controlled access to kernel data structures. The problems enumerated previously are resolved as follows:

  1. Applications do not need to know which kernel binary they are running. The running kernel responds to their request and knows where their data structures are stored. Thus, the correct data structure is always returned or modified.

  2. No time is spent reading or interpreting name lists. Accessing kernel data structures takes only a few system calls.

  3. Sensitive data structures cannot be accessed. The kernel controls the set of data structures that it will return. Nothing else in the kernel is accessible. The kernel can impose its own set of access restrictions on a data structure by data structure basis.

  4. The kernel can use its standard mechanisms for ensuring consistent access to distributed data structures. When requesting process entries, the kernel can acquire the appropriate locks to ensure that a coherent set of data can be returned.

Additional benefits of the interface include the following:

  • The sysctl interface is fully integrated with the jail system call so that processes running in jails can only access those kernel variables that are appropriate for their view of the system.

  • Values to be changed can be validated before the data structure is updated. If modification of the data structure requires exclusive access, an appropriate lock can be obtained before the update is done. Thus, an element can be added to a linked list without danger of another process traversing the list while the update is in progress.

  • Information can be computed only on demand. Infrequently requested information can be computed only when it is requested, rather than being computed continually. For example, many of the virtual-memory statistics are computed only when a system-monitoring program requests them.

  • The interface allows the superuser to change kernel parameters even when the system is running in secure mode (secure mode is described in Section 8.2). To prevent malfeasance, the kernel does not allow /dev/kmem to be opened for writing while the system is running in secure mode. Even when the system is running in secure mode, the sysctl interface will still allow a superuser to modify kernel data structures that do not affect security.

The sysctl system call describes the kernel name space using a management information base (MIB). An MIB is a hierarchical name space much like the filesystem name space, except that each component is described with an integer value rather than with a string name. A hierarchical name space has several benefits:

  • New subtrees can be added without existing applications being affected.

  • If the kernel omits support for a subsystem, the sysctl information for that part of the system can be omitted.

  • Each kernel subsystem can define its own naming conventions. Thus, the network can be divided into protocol families. Each protocol family can be divided into protocol-specific information, and so on.

  • The name space can be divided into those parts that are machine independent and are available on every architecture and those parts that are machine dependent and are defined on an architecture by architecture basis.

Since the addition of the sysctl system call in 4.4BSD, the number of variables it controls has been expanded to include about 1000 values that control the virtual memory system, filesystems, networking stacks, and the underlying hardware, as well as the kernel itself.


Exercises

14.1 What is the purpose of the boot program?

14.2 What is the job of the assembly-language startup? Why is this program written in assembly language?

14.3 What processes are started when the system is booted?

14.4 How are kernel modules loaded into the system at boot time? Give an example of a kernel module.

14.5 The reboot system call causes the system to halt or reboot. Give two reasons why this system call is useful.

14.6 Give two reasons why kernel-loadable modules are useful in developing kernel services. Give one reason not to use them.

14.7 Why is it necessary to have a particular order in which the kernel services are loaded and initialized? Why is it also necessary to have a particular order in which kernel services are shut down?

*14.8 Suppose that a machine does not have a battery-backup time-of-day clock. Propose a method for determining that the time-of-day clock is incorrect. Describe a way to initialize the clock's time of day. What are the limitations of your method?

**14.9 What are the necessary macros to create a kernel-loadable module? How would you test your module without linking it directly into the kernel?


References

Hamby & Mock, 2004. J. Hamby & J. Mock, "FreeBSD Handbook, Chapter 9, Configuring the FreeBSD Kernel," http://www.freebsd.org/doc/en_US.IS08859-l/books/handbook/kernelconfig.html, 2004.


Glossary

absolute pathname
See [pathname]
access control list (ACL)

An access control list replaces the group permissions for a file with a more specific list of the users that are permitted to access the files. The ACL also includes a list of the permissions that each user is granted. These permissions include the traditional read, write, and execute permissions along with other properties such as the right to rename or delete the file.

ACL
See [access control list]
address family

A collection of related address formats, as found in a single communication domain. For example the IPv4 domain uses the Internet address family.

address resolution protocol (ARP)

A communication protocol used to map one network address to another dynamically. For example, ARP is used in FreeBSD to map Internet addresses into Ethernet addresses dynamically.

See also [neighbor discovery]

address translation

A mechanism, typically implemented in hardware, that translates memory addresses supplied by a program into physical memory addresses. This facility is important in supporting multiprogramming because it allows an operating system to load programs into different areas of memory and yet have each program execute as though it were loaded at a single, fixed memory location.

See also [memory-management unit]

advisory lock

A lock that is enforced only when a process explicitly requests its enforcement. An advisory lock is contrasted with a mandatory lock, which is always enforced.

See also [mandatory lock]

ancillary data

Specially interpreted data sent on a network connection. Ancillary data may include protocol-specific data, such as addressing or options.

anonymous object

An anonymous object represents a region of transient backing storage. Pages of an anonymous object are zero-filled on first reference, and modified pages will be stored in the swap area if memory becomes tight. The object is destroyed when no references remain.

ARP
See [address resolution protocol]
association

In the interprocess-communication facilities, a logical binding between two communication endpoints that must be established before communication can take place. Associations may be long lived, such as in stream-based communication, or short lived, such as in a datagram-based communication paradigm.

AST
See [asynchronous system trap]
asymmetric cryptography

A cryptographic system that does not use the same key for decrypting data as it does for encrypting data, sometimes referred to as public key cryptography.

See also [symmetric cryptography]

asynchronous system trap (AST)

A software-initiated interrupt to a service routine. ASTs enable a process to be notified of the occurrence of a specific event asynchronously with respect to its execution. In FreeBSD, ASTs are used to initiate thread rescheduling.

autocontiguration

The probing and identification of hardware attached to the system. Successfully identified hardware is attached to the I/O subsystem. Autoconfiguration is done when the kernel bootstraps itself into operation and any time that a new piece of hardware is attached to the system. In a network protocol, the process by which a system discovers important information about itself and the network (such as its network address and default router) without any help from a user.

background process

In job-control-oriented process-management systems, a process whose process group is different from that of its controlling terminal; thus, this process is currently blocked from most terminal access. Otherwise, a background process is one for which the command interpreter is not waiting—that is, the process was set running with the "&" operator. The opposite of a background process is a foreground process.

backing storage

Storage that holds objects that are removed from main memory during paging and swapping operations.

See also [secondary storage]

block

In the filesystem, a unit of allocation. The filesystem allocates space in block-size units or in fragments of block-size units.

block accounting

The process of maintaining a count of the number of disk blocks available for the storage of new data in the Fast Filesystem.

block size

The natural unit of space allocated to a file (filesystem block size) or the smallest unit of I/O that a character device can do (for disk devices, usually the sector size). In FreeBSD, the filesystem block size is a parameter of the filesystem that is fixed at the time that the filesystem is created.

bootstrapping

The task of bringing a system up into an operational state. When a machine is first powered on, it is typically not running any program. Bootstrapping initializes the machine, loads a program from secondary storage into main memory, and sets that program running.

bottom half

With regard to system operation, the collection of routines in the kernel that is invoked as a result of interrupts. These routines cannot depend on any per-process state.

See also [top half]

breakpoint fault

A hardware trap that is generated when a process executes a breakpoint instruction.

broadcast

A transmission to all parties. In a network, a broadcast message is transmitted to all stations attached to a common communication medium.

bss segment

The portion of a program that is to be initialized to zero at the time the program is loaded into memory. The name bss is an abbreviation for "block started by symbol."

See also [data segment]
See also [stack segment]
See also [text segment]

buffer cache

A cache of recently used disk blocks. In FreeBSD, the buffer cache has been merged with the virtual-memory cache.

buffered

As in "buffered I/O"; a technique whereby data are held, or buffered, to minimize the number of I/O operations that are done. For example, the standard I/O library buffers output to files by accumulating data to be written until there is a full filesystem block to write or until the application requests that the data be flushed to disk.

bus

A standardized electrical and mechanical interconnection for components of a computer.

byte

A unit of measure applied to data. A byte is almost always 8 bits.

See also [octet]

callback

In the kernel, a mechanism to notify a subsystem that an asynchronous operation has completed. In NFS, a scheme where a server keeps track of all the objects that each of its clients has cached. When a cached object is held by two or more clients and one of them modifies it, the server sends an eviction notice to all the other clients holding that object so that they can purge it from their cache.

See also [eviction notice]
See also [lease]

callout queue

The kernel data structure that describes waiting events. Each event in the queue is described by a structure that contains a function to be called, a pointer to be passed as an argument to the function, and the number of clock ticks until the event should occur.

canonical mode

A terminal mode. Characters input from a terminal or a pseudo-terminal that is running in canonical mode are processed to provide standard line-oriented editing functions, and input is presented to a process on a line-by-line basis. When the terminal is processing in noncanonical mode, input is passed through to the reading process immediately and without interpretation. Canonical mode is also known as cooked mode, and noncanonical mode is also known as raw mode.

caught signal

A signal the delivery of which to a process results in a signal-handler procedure being invoked. A signal handler is installed by a process with the sigaction system call.

C-block

The buffer that holds the actual data in a C-list data structure.

central processing unit (CPU)

The primary computational unit in a computer. The CPU is the processing unit that executes applications. A multiprocessor will have more than one CPU. Other processing units may be present in a computer—for example, for handling I/O.

character

A datum that represents a single printable or control symbol. Characters are usually 8 or 16 bits long.

See also [byte]
See also [octet]

character-at-a-time mode

A mode of operation for a pseudo-terminal device whereby processes reading from the pseudo-terminal receive input immediately as it is typed. This mode differs from raw mode in that certain input processing, such as interpreting the interrupt character, is still performed by the system.

See also [canonical mode]

character device

A device that provides either a character-stream-oriented I/O interface or, alternatively, an unstructured (raw) interface. All devices in FreeBSD use the character-device interface.

character-device interface

The conventions established for accessing devices within the kernel. These conventions include the set of procedures that can be called to do I/O operations and the parameters that must be passed in each call.

checksum

The value of a mathematical function computed for a block of data; used to detect corruption of the data block.

child process

A process that is a direct descendant of another process as a result of being created with a fork system call.

client process

In the client-server model of communication, a process that contacts a server process to request services. A client process is usually unrelated to a server process; the client's only association with the server is through a communication channel.

See also [server process]

C-list

A linked-list data structure, used by the system in supporting serial line I/O.

cloning route

A routing entry that is not used directly but that causes a new instance of a route to be created. For example, the route to a local Ethernet is set up as a cloning route so that individual host routes will be created for each local host when referenced.

cluster

The logical grouping of contiguous pages of virtual memory or a file. In FreeBSD, this grouping is used by the kernel to aggregate pages together when writing or reading them to or from the disk to reduce the number of I/O operations needed to move data in and out of memory.

cold start

The initial phase of a bootstrap procedure. The term is derived from the fact that the software assumes nothing about the state of the machine—as though the machine had just been turned on and were cold.

communication domain

An abstraction used by the interprocess-communication facilities to organize the properties of a communication network or similar facility. A communication domain includes a set of protocols, termed the protocol family; rules for manipulating and interpreting names; the address family; and, possibly, other intrinsic properties. The facilities provided by the system for interprocess communication are defined such that they are independent of the communication domains supported by the system. This design makes it possible for applications to be written in a communication-domain-independent manner.

communication protocol

A set of conventions and rules used by two communicating processes. Communication protocols are most often associated with networking.

configuration file

A file that contains parameters for the system-configuration program /usr/sbin/config. This file describes the device drivers that should be configured into the kernel and other basic kernel functionality such as the enabling of symmetric multiprocessing support.

connect request

A request passed to the user-request routine of a communication-protocol module as a result of a process making a connect system call on a socket. The request causes the system to attempt to establish an association between a local and a remote socket.

context switching

The action of interrupting the currently running thread and of switching to another thread. Context switching occurs as one thread after another is scheduled for execution. An interrupted thread's context is saved in that thread's thread control block, and another thread's context is loaded.

continuation style

A style of programming where two or more functions operate cooperatively by calling each other instead of returning directly at the end of execution. When the currently executing function is done with its work it calls another function, which was passed as one of the first function's arguments, as part of its return() call. Programming with continuations has the effect of creating a function chain and is often used when a system wants to submit work to a hardware coprocessor but wishes to have a cleanup routine called by the hardware coprocessor when the coprocessor has completed the job.

continue signal

Signal 19 (SIGCONT). A signal that, when delivered to a stopped or sleeping process, causes that process to resume execution.

controlling process

The session leader that established the connection to the controlling terminal.

See also [session leader]

controlling terminal

The pseudo-terminal associated with a process's session from which keyboard-related signals may be generated. The controlling terminal for a process is normally inherited from the process's parent.

control request

A request passed to a communication-protocol module as a result of a process making an ioctl or setsockopt system call on a socket.

cooked mode
See [canonical mode]
copy-on-write

A technique whereby multiple references to a common object are maintained until the object is modified (written). Before the object is written, a copy is made; the modification is made to the copy rather than to the original. In virtual-memory management, copy-on-write is a common scheme that the kernel uses to manage pages shared by multiple processes. All the page-table entries mapping a shared page are set such that the first write reference to the page causes a page fault. When the page fault is serviced, the faulted page is replaced with a private copy, which is writable.

core file

A file (named procname.core) that is created by the system when certain signals are delivered to a process. The file contains a record of the state of the process at the time the signal occurred. This record includes the contents of the process's virtual address space and, on most systems, the user structure.

CPU
See [central processing unit]
crash

Among computer scientists, an unexpected system failure.

crash dump

A record of the state of a machine at the time of a crash. This record is usually written to a place on secondary storage that is thought to be safe so that it can be saved until the information can be recovered.

credential

A structure that identifies a user. It contains the real, effective, and saved user and group identifiers.

See also [real user identifier]
See also [real group identifier]
See also [effective user identifier]
See also [effective group identifier]
See also [saved UID]
See also [saved GID]

current working directory

The directory from which relative pathnames are interpreted for a process. The current working directory for a process is set with the chdir or fchdir system call.

cylinder group

In the Fast Filesystem, a collection of blocks on a disk drive that is grouped together to use localizing information. That is, the filesystem allocates inodes and data blocks on a per-cylinder-group basis. Cylinder group is a historic name from the days when the geometry of disks was known.

daemon

A long-lived process that provides a system-related service. There are daemon processes that execute in kernel mode (e.g., the pagedaemon) and daemon processes that execute in user mode (e.g., the routing daemon). The Old English term daemon means "a deified being," as distinguished from the term demon, which means "an evil spirit."

DARPA

Defense Advanced Research Projects Agency. An agency of the U.S. Department of Defense that is responsible for managing defense-sponsored research in the United States.

datagram socket

A type of socket that supports an unreliable message transport that preserves message boundaries.

data segment

The segment of a process's address space that contains the initialized and uninitialized data portions of a program.

See also [bss segment]
See also [stack segment]
See also [text segment]

decapsulation

In network communication, the removal of the outermost header information from a message.

See also [encapsulation]

demand paging

A memory-management technique in which memory is divided into pages and the pages are provided to processes as needed—that is, on demand.

See also [pure demand paging]

demon
See [daemon]
denial of service attack

Any attempt to overload a system such that it is unable to do work for legitimate users of the system. For example, sending a system so many packets that it runs out of mbufs and so cannot process any other network traffic.

descriptor

An integer assigned by the system when a file is referenced by the open system call or when a socket is created with the socket, pipe, or socketpair system calls. The integer uniquely identifies an access path to the file or socket from a given process or from any of that process's children. Descriptors can also be duplicated with the dup and fcntl system calls.

descriptor table

A per-process table that holds references to objects on which I/O may be done. I/O descriptors are indices into this table.

device

In UNIX, a peripheral connected to the CPU.

device driver

A software module that is part of the kernel and that supports access to a peripheral device.

device flags

Data specified in a system configuration file and passed to a device driver. The use of these flags varies across device drivers. Device drivers for terminal devices use the flags to indicate the terminal lines on which the driver should ignore modem-control signals on input.

device number

A number that uniquely identifies a device within the character-device class. Historically, a device number comprises two parts: a major device number and a minor device number. In FreeBSD 5.2, device numbers are assigned dynamically and are used only for backward compatibility with older applications.

device special file

A file through which processes can access hardware devices on a machine. For example, a sound card is accessed through such a file.

direct memory access (DMA)

A facility whereby a peripheral device can access main memory without the assistance of the CPU. DMA is typically used to transfer contiguous blocks of data between main memory and a peripheral device.

directory

In UNIX, a special type of file that contains entries that are references to other files. By convention, a directory contains at least two entries: dot (.) and dot-dot (..). Dot refers to the directory itself; dot-dot refers to the parent directory.

directory entry

An entry that is represented by a variable-length record structure in the directory file. Each structure holds an ASCII string that represents the filename, the number of bytes of space provided for the string, the number of bytes of space provided for the entry, the type of the file referenced by the entry, and the number of the inode associated with the filename. By convention, a directory entry with a zero inode number is treated as unallocated, and the space held by the entry is available for use.

directory table

The top level of a two-level hierarchy of data structures used by a forward-mapped page-table algorithm to describe the virtual address space of a process. Each entry in a directory-table points to a page of page-table pages. A two-level mapping hierarchy is used on the PC architectures.

See also [forward-mapped page table]
See also [page-table entry]
See also [page-table pages]

dirty

In computer systems, modified. A system usually tracks whether an object has been modified—is dirty—because it needs to save the object's contents before reusing the space held by the object. For example, in the virtual-memory system, a page in the virtual-memory cache is dirty if its contents have been modified. Dirty pages must be written to the swap area or filesystem before they are reused.

disk partition

A software scheme that divides a disk drive into one or more linear extents or partitions. Each partition is a contiguous region of a disk drive that is used as a swap area or to hold a filesystem.

distributed program

A program that is partitioned among multiple processes, possibly spread across multiple machines.

DMA
See [direct memory access]
double indirect block
See [indirect block]
effective GID
See [effective group identifier]
effective group identifier (effective GID)

The first entry in the groups array. The effective GID, along with the other GIDs in the groups array, is used by the filesystem to check group access permission. The effective GID is set when a set-group-identifier program is executed. saved group identifier.

See also [credential]
See also [group identifier]
See also [real group identifier]

effective UID
See [effective user identifier]
effective user identifier (effective UID)

The UID that the system uses to check many user permissions. For example, the effective UID is used by the filesystem to check owner access permission on files. The effective UID is set when a set-user-identifier program is executed. saved user identifier

See also [credential]
See also [real user identifier]
See also [user identifier]

elevator sorting algorithm

An algorithm used by the device drivers for I/O requests for moving head disks. The algorithm sorts requests into a cyclic ascending order based on the block number of the request. The name is derived from the fact that the algorithm orders disk requests in a manner similar to the way ride requests for an elevator would be handled most efficiently.

emulate

To simulate. FreeBSD can emulate the system-call interface of other variants of the UNIX operating system. For example, FreeBSD can run binaries compiled for Linux.

encapsulation

In network communication, the procedure by which a message is created that has an existing message enclosed in it as data. A protocol normally encapsulates a message by crafting a leading protocol header that indicates that the original message is to be treated as data.

See also [decapsulation]

erase character

The character that is recognized by the terminal handler, when the latter is running in canonical mode, to mean "delete the last character in the line of input." Each terminal session can have a different erase character, and that erase character can be changed at any time with a tcsetattr system call. The terminal handler does not recognize the erase character on terminals that are in noncanonical mode.

See also [kill character]
See also [word-erase character]

errno

The global variable in C programs that holds an error code that indicates why a system call failed. The value to be placed in errno is returned by the kernel in the standard return register; it is moved from this return register to errno by code in the C run-time library.

event handler

A function that is registered with a software system that is to be called at a later time when a particular event occurs.

See also [callout queue]
See also [kqueue]

eviction notice

A callback message from a server to a client notifying the client that its lease for an object is being terminated. A lease is usually terminated because another client wants to modify the object that the lease represents.

See also [callback]
See also [lease]

extension header

Message header that can be easily added and removed from a packet because it contains its length and some indication of where and how to begin processing the next header, if such exists.

fault rate

The rate at which a process generates page faults. For a reference string, the fault rate is defined to be time independent by its being specified as the number of page faults divided by the length of the reference string.

fetch policy

The policy used by a demand-paged virtual-memory-management system in processing page faults. Fetch policies differ primarily in the way that they handle prepaging of data.

fifo file

In the filesystem, a type of file that can be used for interprocess communication. Data written by one process to a fifo are read by another in the order in which they were sent. The name refers to the fact that data are transferred in a first-in, first-out fashion.

file

An object in the filesystem that is treated as a linear array of bytes. A file has at least one name, and it exists until all its names are deleted explicitly.

file entry
See [file structure]
file handle

A globally unique token created by an NFS server and passed back to an NFS client. The client can then use the file handle to refer to the associated file on the server. A handle is created when a file is first opened; it is passed to the server by the client in later operations, such as read and write, that reference the open file.

filename

A string of ASCII characters that names an ordinary file, special file, or directory. The characters in a filename cannot include null (0) or the ASCII code for slash (/).

file offset

A byte offset associated with an open file descriptor. The file offset for a file descriptor is set explicitly with the lseek system call, or implicitly as a result of a read or write system call.

file structure

The data structure used by the kernel to hold the information associated with one or more open file descriptors that reference a file. Usually, each open file descriptor references a unique file structure. File structures may be shared, however, when open descriptors are duplicated with the dup and dup2 system calls, inherited across a fork system call, or received in a message through the interprocess-communication facilities.

filesystem

A collection of files. The UNIX filesystem is hierarchical, with files organized into directories. Filesystems include facilities for naming files and for controlling access to files. A filesystem resides on a single logical device that may be built from part of a single disk drive or from a set of disk drives that have been aggregated together.

fill-on-demand page fault

The first page fault for an individual page; it must be resolved by retrieval of data from the filesystem or by allocation of a zero-filled page.

first-level bootstrap

The initial code that is executed in a multilevel bootstrapping operation. Usually, the first-level bootstrap is limited in size and does little more than bootstrap into operation a larger, more intelligent program. Typically, the first-level bootstrap loads the /boot program so that /boot can, in turn, bootstrap the kernel.

foreground process

In job-control-oriented process-management systems, a process whose process group is the same as that of its controlling terminal; thus, the process is allowed to read from and to write to the terminal. Otherwise, a foreground process is one for which the command interpreter is currently waiting. The opposite of a foreground process is a background process.

forward

The direction a network packet takes through a system if it is received by a host for which it is not, ultimately, destined.

See also [inbound]
See also [router]

forward-mapped page table

A large, contiguous array indexed by the virtual address that contains one element, or page-table entry, for each virtual page in the address space. This element contains the physical page to which the virtual page is mapped, as well as access permissions and status bits telling whether the page has been referenced or modified, and a bit showing whether the entry contains valid information. Most current memory-management-unit designs for architectures with 32-bit address spaces use some variant of a forward-mapped page table.

See also [inverted page table]
See also [memory-management unit]

fragment

In the filesystem, a part of a block. The filesystem allocates new disk space to a file as a full block or as one or more fragments of a block. The filesystem uses fragments, rather than allocating space in only full block-size units, to reduce wasted space when the size of a full block is large.

fragment-descriptor table

A data structure in the Fast Filesystem that describes the fragments that are free in an entry of the allocation map. The filesystem uses the fragment-descriptor table by taking a byte in the allocation map and using the byte to index into the fragment-descriptor table. The value in the fragment-descriptor table indicates how many fragments of a particular size are available in the entry of the allocation map. By doing a logical AND with the bit corresponding to the desired fragment size, the system can determine quickly whether a desired fragment is contained within the allocation-map entry.

free list

In the memory-management system, the list of available pages of physical memory (also called the memory free list). There is a similar free list in the system for dynamically allocated kernel memory. Many kernel data structures are dynamically allocated, including vnodes, file-table entries, and disk-quota structures.

free-space reserve

A percentage of space in a filesystem that is held in reserve to ensure that certain allocation algorithms used by the filesystem will work well. By default, 8 percent of the available space in the Fast Filesystem is held in reserve.

gateway
See [router]
generation number

The number assigned to an inode each time that the latter is allocated to represent a new file. Each generation number is used only once. To make file handles harder to guess, most NFS implementations, including FreeBSD, use a random-number generator to select a new generation number.

GID
See [group identifier]
global page-replacement algorithm

An algorithm that does page replacement according to systemwide criteria. A global-page-replacement strategy tends to make the most efficient use of the system memory. However, a single process can thrash the entire system by trying to use all the available memory.

group identifier (GID)

An integer value that uniquely identifies a collection of users. GIDs are used in the access-control facilities provided by the filesystem. saved group identifier

See also [credential]
See also [effective group identifier]
See also [real group identifier]
See also [set-group-identifier program]

half-open connection

A connection that is open for communication in one direction between two endpoints. For example, a client may close its sending side of a stream connection because it has no more data to send but leave its receiving half of the connection open so that it can continue to receive data from the server.

handler

A procedure that is invoked in response to an event such as a signal.

hard limit

A limit that cannot be exceeded.

See also [soft limit]

hard link

A directory entry that directly references an inode. If there are multiple hard links to a single inode and if one of the links is deleted, the remaining links still reference the inode. By contrast, a symbolic link is a file that holds a pathname that references a file.

See also [symbolic link]

header prediction

A heuristic used by TCP on incoming packets to detect two common cases: the next expected data segment for an existing connection or an acknowledgment plus a window update for one or more data segments. When one of these two cases arises, and the packet has no additional flags or state indications, the fully general TCP input processing is skipped.

heap

The region of a process that can be expanded dynamically with the sbrk system call (or malloc() C library call). The name is derived from the disorderly fashion in which data are placed in the region.

high watermark

An upper bound on the number of data that may be buffered. In the interprocess-communication facilities, each socket's data buffer has a high watermark that specifies the maximum number of data that may be queued in the data buffer before a request to send data will block the process (or will return an error if nonblocking I/O is being used).

See also [low watermark]

hole

In a file, a region that is part of the file but that has no associated data blocks. The filesystem returns zero-valued data when a process reads from a hole in a file. A hole is created in a file when a process positions the file pointer past the current end-of-file, writes some data, and then closes the file. The hole appears between the previous end-of-file and the beginning of the newly written data.

home directory

The current working directory that is set for a user's shell when the user logs into a system. This directory is usually private to the user. The home directory for a user is specified in a field in the password-file entry for the user.

hop limit

The number of routers through which a network packet may be forwarded before it is dropped.

See also [router]

host-unreachable message

A network layer error message that indicates that the host to which a previous message was directed is unavailable because there is no known path to the desired host.

ICMP
See [Internet control message protocol]
idempotent

An operation that can be repeated several times without changing the final result or causing an error. For example, writing the same data to the same offset in a file is idempotent, because it will yield the same result whether it is done once or many times. However, trying to remove the same file more than once is nonidempotent because the file will no longer exist after the first try.

idle loop

The block of code inside the kernel that is executed when there is nothing else to run. In FreeBSD, the idle loop zeros pages on the free list while it waits for a thread to be added to the run queue.

idle queue

The idle queue is where all idle threads are stored. An idle thread is run only when a CPU has nothing else to do.

See also [run queue]
See also [sleep queue]

inbound

The direction a network packet is traveling if it has reached the system for which it is destined. An inbound network packet is delivered to an application on a system or causes an error if no appropriate application is found.

See also [forward]

indirect block

In the filesystem, an auxiliary data block that holds the number of a data block. The first 12 blocks of a file are pointed to directly by the inode. Additional data blocks are described with a pointer from the inode to an indirect data block; the system must first fetch the indirect block that holds the number of the data block. In FreeBSD, the kernel may have to fetch as many as three indirect blocks to locate the desired data block. An indirect block that contains data-block numbers is termed a single-level indirect block; an indirect block that contains block numbers of single-level indirect blocks is called a double-level indirect block; an indirect block that contains block numbers of double-level indirect blocks is called a triple-level indirect block.

init

The first user program (/sbin/init) that runs when the system is booted.

initial sequence number
See [sequence space]
inode

A data structure used by the filesystem to describe a file. The contents of an inode include the file's type and size, the UID of the file's owner, the GID of the directory in which it was created, and a list of the disk blocks and fragments that make up the file. Note that inodes do not have names; directory entries are used to associate a name with an inode.

input/output (I/O)

The transfer of data between the computer and its peripheral devices.

integrity-check value

A value computed over a range of data by a sender that is used by a receiver to ensure that data transmitted across a network was not corrupted.

See also [checksum]

interactive program

A program that must periodically obtain user input to do its work. A screen-oriented text editor is an example of an interactive program.

Internet control message protocol (ICMP)

A communication protocol used for reporting errors and controlling the operation of the Internet protocols. Each of IPv4 and IPv6 includes its own version of ICMP, called ICMPv4 and ICMPv6, respectively.

Internet key exchange

A network protocol for exchanging keys used in the IPSec security protocols.

Internet protocol

See IPv4.

interpreter

A program that parses and executes a descriptive language in a single step, rather than using the more common two-stage process of compiling the language and executing the resulting binary. The shell is an example of an interpreter; it parses and executes a shell script rather than first compiling it.

interprocess communication (IPC)

The transfer of data between processes. Most facilities for interprocess communication are designed such that data are transferred between objects other than processes. An interprocess-communication model that is not directly process oriented is advantageous because it is possible to model scenarios in which communication endpoints are location independent and, possibly, are migrated dynamically. For example, in FreeBSD, communication is between sockets rather than between processes.

interrupt

In computer systems, an event external to the currently executing process that causes a change in the normal flow of instruction execution. Interrupts usually are generated by hardware devices that are external to the CPU.

interrupt stack

A run-time stack that is used by procedures that are invoked to respond to interrupts and traps. In FreeBSD 5.2, each device has its own thread whose context includes an interrupt stack.

inverted page table (reverse-mapped page table)

A hardware-maintained memory-resident table that contains one entry per physical page and that is indexed by physical address instead of by virtual address. An entry contains the virtual address to which the physical page is currently mapped; the entry also includes protection and status attributes. The hardware does virtual-to-physical address translation by computing a hash function on the virtual address to select an entry in the table. The hardware handles collisions by linking together table entries and making a linear search of this chain until it finds the matching virtual address.

See also [forward-mapped page table]

I/O
See [input/output]
I/O redirection

The redirection of an I/O stream from the default assignment. For example, all the standard shells permit users to redirect the standard output stream to a file or process. I/O redirection is implemented in the shell by first closing the descriptor associated with the I/O stream and then opening or duplicating another descriptor in its place.

I/O stream

A stream of data directed to, or generated from, a process. Most I/O streams in UNIX have a single common data format that permits users to write programs in a tool-oriented fashion and to combine these programs in pipelines by directing the standard output stream of one program to the standard input stream of another.

iovec

A data structure used to specify user I/O requests made to the kernel. Each structure holds the address of a data buffer and the number of bytes of data to be read or written. Arrays of such structures are passed to the kernel in readv and writev system calls.

See also [scatter-gather I/O]

I/O vector
See [iovec]
IPC
See [interprocess communication]
IPSec

The set of protocols that implement network layer security in the Internet protocols, versions 4 and 6.

IPv4 domain

Version 4 of the Internet protocols. IPv4 used to be called the Internet protocols until version 6 was developed.

See also [IPv6 domain]

IPv6 domain

Version 6 of the Internet protocols. The newest version of the Internet protocols with support for larger addresses, security, and autoconfiguration.

See also [IPv4 domain]

job

In UNIX, a set of processes that all have the same process-group identifier. Jobs that have multiple processes are normally created with a pipeline. A job is the fundamental object that is manipulated with job control.

job control

A facility for managing jobs. With job control, a job may be started, stopped, and killed, as well as moved between the foreground and the background. The terminal handler provides facilities for automatically stopping a background job that tries to access the controlling terminal and for notifying a job's controlling process when such an event occurs.

keepalive packet

A type of packet used by TCP to maintain information about whether a destination host is up. Keepalive packets are sent to a remote host, which, if it is up, must respond. If a response is not received in a reasonable time to any of several keepalive packets, then the connection is terminated. Keepalive packets are used on only those TCP connections that have been created for sockets that have the SO_KEEPALIVE option set on them.

keepalive timer

A timer used by the TCP protocol when using keepalive packets. The timer is set when a keepalive packet is transmitted. If a response to the packet is not received before the timer expires several times, then the connection is shut down.

kernel

The central controlling program that provides basic system facilities. The FreeBSD kernel creates and manages processes, provides functions to access the filesystem, and supplies communication facilities. The FreeBSD kernel is the only part of FreeBSD that a user cannot replace.

kernel loadable modules

A collection of software that implements a kernel service but that is not statically linked into the kernel's image. Kernel-loadable modules are brought into the system dynamically, possibly at run time, by actions initiated either by the system or a user. See also permanent kernel modules.

kernel mode

The most privileged processor-access mode. The FreeBSD kernel operates in kernel mode.

See also [user mode]

kernel process

A process that executes with the processor in kernel mode. The pagedaemon and swapper processes are examples of kernel processes.

kernel state

The run-time execution state for the kernel. This state, which includes the program counter, general-purpose registers, and run-time stack, must be saved and restored on each context switch.

key

In the kernel, a piece of data that uniquely identifies some resource in the system. When used by an interprocess communication system, it identifies an endpoint of communication such as a message queue or a shared facility like a region of shared memory.

kill character

The character that is recognized by the terminal handler in canonical mode to mean "delete everything typed on this terminal after the most recent end-of-line character." Each terminal session can have a different kill character, and the user can change that kill character at any time with an tcsetattr system call. The terminal handler does not recognize the kill character on terminals that are in noncanonical mode.

See also [erase character]
See also [word-erase character]

kqueue

A kernel data structure associated with a file descriptor that describes a set of events for which the process referencing the descriptor wants to be notified. Events include both dynamic transitions, such as the arrival of data to read, and state transitions, such as a rename of the file associated with the descriptor.

lease

A ticket permitting an activity that is valid until a specified expiration time. In the NQNFS protocol, a client gets a lease from its server to read, write, or read and write a file. As long as the client holds a valid lease, it knows that the server will notify it if the file status changes. Once the lease has expired, the client must contact the server to request a new lease before using any data that it has cached for the file.

See also [callback]
See also [eviction notice]

least-recently used (LRU)

A policy of reuse whereby the least-recently used items are reused first. For example, in the filesystem, there is a fixed number of vnodes available for accessing files. Vnodes that hold valid file data are reallocated in an LRU order, in the hope that the file referenced by the vnode may be reused by a later open request.

line discipline

A processing module in the kernel that provides semantics for an asynchronous serial interface or for a software emulation of such an interface. Line disciplines are described by a procedural interface whose entry points are stored in the linesw data structure.

line mode
See [canonical mode]
link layer

Layer 2 in the ISO Open Systems Interconnection Reference Model. In this model, the link layer is responsible for the (possibly unreliable) delivery of messages within a single physical network. The link layer corresponds most closely to the network-interface layer of the FreeBSD network subsystem.

See also [network-interface layer]

listen request

A request passed to a communication-protocol module as a result of a process making a listen system call on a socket. This request indicates that the system should listen for requests to establish a connection to the socket. Otherwise, the system will reject any connection requests that it receives for the socket.

load average

A measure of the CPU load on the system. The load average in FreeBSD is an average of the number of processes ready to run or waiting for short-term events such as disk I/O to complete, as sampled once per second over the previous one-minute interval of system operation.

local domain

A communication domain in the interprocess-communication facilities that supports stream- and datagram-oriented styles of communication between processes on a single machine.

locality of reference

A phenomenon whereby memory references of a running program are localized within the virtual address space over short periods. Most programs tend to exhibit some degree of locality of reference. This locality of reference makes it worthwhile for the system to prefetch pages that are adjacent to a page that is faulted, to reduce the fault rate of a running program.

local page-replacement algorithm

An algorithm for page replacement that first chooses a process from which to replace a page and then chooses a page within that process based on per-process criteria. Usually, a process is given a fixed number of pages and must then select from among its own pages when it needs a new page.

logical block

A block defined by dividing a file's linear extent by the underlying filesystem block size. Each logical block of a file is mapped into a physical block. This additional level of mapping permits physical blocks to be placed on disk without concern for the linear organization of the logical blocks in a file.

long-term-scheduling algorithm
See [short-term-scheduling algorithm]
low watermark

A lower bound that specifies the minimum number of data that must be present before an action can be taken. In the interprocess-communication facilities, each socket's data buffer has a low watermark that specifies the minimum number of data that must be present in the data buffer before a reception request will be satisfied.

See also [high watermark]

LRU
See [least-recently used]
MAC
See [mandatory access control]
machine check

An exceptional machine condition that indicates that the CPU detected an error in its operation. For example, a machine check is generated if a parity error is detected in a cache memory.

magic number

The number located in the first few bytes of an executable file that specifies the type of the executable file. Many on-disk data structures have a magic number to help verify their contents.

main memory

The primary memory system on a machine.

major device number

A number that uniquely identifies a device within the character-device class. Historically, the major device number was used as an index into the kernel's device table. In FreeBSD 5.2, device numbers are assigned dynamically and are used only for backward compatibility with older applications.

mandatory access control (MAC)

Mandatory access controls are labels on files that allow additional restrictions to be placed on their use. For example, a file labelled "secret" may only be shown to users with a clearance level of secret or higher.

See also [access control list]

mandatory lock

A lock that cannot be ignored or avoided. A mandatory lock is contrasted with an advisory lock, which is enforced only when a process explicitly requests its enforcement.

See also [advisory lock]

mapped object

An object whose pages are mapped into a process address space. Processes map objects into their virtual address space using the mmap system call.

mapping structure

The machine-dependent state required to describe the translation and access rights of a single page.

See also [page-table entry]

marshalling

Preparing a set of parameters to be sent across a network. Marshalling includes replacing pointers by the data to which they point and converting binary data to the canonical network byte order.

See also [remote procedure call]

masked signal

A signal blocked in a sigprocmask system call. When a signal is masked, its delivery is delayed until it is unmasked. In addition, in FreeBSD, the system automatically masks a caught signal while that signal is being handled.

master device
See [slave device]
maximum segment lifetime (MSL)

The maximum time that a segment of data may exist in the network.

See also [2MSL timer]

mbuf

A data structure that describes a block of data. Mbufs are used in the interprocess-communication facilities for holding network packets, as well as data that are internal to the network protocol modules. "Mbuf" is shorthand for "memory buffer."

memory address

A number that specifies a memory location. Memory addresses are often categorized as physical or virtual according to whether they reference physical or virtual memory.

memory free list
See [free list]
memory-management system

The part of the operating system that is responsible for the management of memory resources available on a machine.

memory-management unit (MMU)

A hardware device that implements memory-management-related tasks, such as address translation and memory protection. Most contemporary memory-management units also provide support for demand-paged virtual-memory management.

See also [address translation]

message queue

A local interprocess communication mechanism that supports in-order delivery of messages. Messages are inserted at one end of the queue and removed from the other and the kernel guarantees their ordering.

metadata

In filesystems, metadata provides pointers and descriptions for linking multiple disk sectors into files and identifying those files. Metadata are the directories, inodes, and free block maps that give structure to raw storage capacity.

minor device number

An integer number that uniquely identifies a subunit of a device. The minor device number is interpreted on a per-device basis. For example, the minor device number for a disk device specifies a subunit termed a partition, whereas the minor device number for a pseudo-terminal device identifies a specific master-slave pair.

MMU
See [memory-management unit]
modem control

For data-communication equipment, the support of a set of signals used to ensure reliable initiation and termination of connections over asynchronous serial lines, defined by the RS-232 standard. Support for modem control is normally important for only serial lines that are accessed via dialup modems.

MSL
See [maximum segment lifetime]
multilevel feedback queue

A queueing scheme in which requests are partitioned into multiple prioritized subqueues, with requests moving between subqueues based on dynamically varying criteria. The FreeBSD kernel uses a multilevel-feedback-queueing scheme for scheduling the execution of threads.

neighbor discovery

The technique by which a system on a network discovers the hardware address of its neighbors, including its router, so that it can send network packets to them. The neighbor discovery protocol is used by IPv6.

See also [address resolution protocol]

network architecture

The collection of protocols, facilities, and conventions (such as the format of a network address) that defines a network. Like machine architectures, network architectures may be realized in different ways. For example, some network architectures are specifically designed to permit their implementation in hardware devices.

network byte order

The order defined by a network for the transmission of protocol fields that are larger than one octet. In IPv4 and IPv6, this order is "most significant octet first."

network-interface layer

The layer of software in the FreeBSD network subsystem that is responsible for transporting messages between hosts connected to a common transmission medium. This layer is mainly concerned with driving the transmission media involved and with doing any necessary link-level protocol encapsulation and decapsulation.

See also [link layer]

network layer

The layer of software in the FreeBSD network subsystem that is responsible for implementing ISO layer 2 functionality. In the IPv4 domain this functionality is implemented in the IP protocol module.

network mask

A value that is used in the Classless Inter-Domain Routing (CIDR) addressing scheme of the Internet. A network mask specifies which bits should be considered significant when extracting the network part from an address.

newbus

The device-driver infrastructure used in FreeBSD to manage the devices on the system. Newbus includes machine-independent routines and data structures for use by machine-dependent layers and provides a framework for dynamic allocation of data structures for each device. See also autoconfiguration.

nice

A user-controllable process-scheduling parameter. The value of a process's nice variable is used in calculating the scheduling priority of the process's threads. Positive values of nice mean that the process is willing to receive less than its share of the processor. Negative values of nice mean that the process requests more than its share of the processor.

nonblocking I/O

A mode in which a descriptor may be placed, whereby the system will return an error if any I/O operation on the descriptor would cause the process to block. For example, if a read system call is done on a descriptor that is in nonblocking I/O mode, and no data are available, the system will return the error code EAGAIN, rather than block the process until data arrive.

See also [polling I/O]
See also [signal-driven I/O]

noncaching lease

A noncaching lease is issued by an NFS server to a client to require that the client do all file operations synchronously with the server.

See also [read-caching lease]
See also [write-caching lease]

noncanonical mode
See [canonical mode]
nonlocal goto

A transfer in control that circumvents the normal flow of execution in a program across routine boundaries. For example, if procedure A calls procedure B, and B calls C, then a direct transfer of control from C back to A (bypassing B) would be a nonlocal goto.

nonresident object

An object that is not present in main memory. For example, a page in the virtual address space of a process may be nonresident if it has never been referenced.

object
See [virtual-memory object]
octet

A basic unit of data representation; an 8-bit byte. The term octet is used instead of byte in the definition of many network protocols because some machines use other byte sizes.

optimal replacement policy

A replacement policy that optimizes the performance of a demand-paging virtual-memory system. In this book, a policy whereby the full reference string of a program is known in advance, and pages are selected such that the number of page faults is minimized.

orphaned process group

A process group in which the parent of every member is either itself a member of the group or is not a member of the group's session. Such a parent would normally be a job-control shell capable of resuming stopped child processes.

out-of-band data

Data transmitted and received out of the normal flow of data. Stream sockets support a logically separate out-of-band data channel through which at least one message of at least 1 octet of data may be sent. The system immediately notifies a receiving process of the presence of out-of-band data, and out-of-band data may be retrieved from the stream out of the order in which normal data are received.

overlay

In computer systems, a region of code or data that may be replaced with other such regions on demand. Overlays are usually loaded into a process's address space on demand, possibly on top of another overlay. Overlays are a commonly used scheme for programs that are too large to fit in the address space of a machine that does not support virtual memory.

page

In memory management, the fixed-sized unit of measure used to divide a physical or virtual address space.

See also [demand paging]

pagedaemon

In FreeBSD, the name of the kernel process that is responsible for writing parts of the address space of a process to secondary storage, to support the paging facilities of the virtual-memory system.

See also [swapper]

page fault

An exception generated by a process's reference to a page of that process's virtual address space that is not marked as resident in memory.

pagein

An operation done by the virtual-memory system in which the contents of a page are read from secondary storage.

pageout

An operation done by the virtual-memory system in which the contents of a page are written to secondary storage.

pager

A kernel module responsible for providing the data to fill a page and for providing a place to store that page when it has been modified and the memory associated with it is needed for another purpose.

page reclaim

A page fault, where the page that was faulted is located in memory, usually on the inactive or cache list.

page-table entry (PTE)

The machine-dependent data structure that identifies the location and status of a page of a virtual address space. When a virtual page is in memory, the PTE contains the page-frame number that the hardware needs to map the virtual page to a physical page.

page-table pages

The top level of a two-level hierarchy of data structures used by a forward-mapped page-table algorithm to describe the virtual address space of a process. On the PC, page-table pages are stored in an array called the directory table; each entry in a page-table page points to a page of bottom-level page-table entries.

See also [directory table]
See also [forward-mapped page table]
See also [page-table entry]

paging

The actions of bringing pages of an executing process into main memory when they are referenced and of removing them from memory when they are replaced. When a process executes, all its pages are said to reside in virtual memory. Only the actively used pages, however, need to reside in main memory. The remaining pages can reside on disk until they are needed.

panic

In UNIX, an unrecoverable system failure detected by the kernel. FreeBSD automatically recovers from a panic by rebooting the machine, repairing any filesystem damage, and then restarting normal operation.

See also [crash dump]

parent process

A process that is a direct relative of another process as a result of a fork system call.

partition
See [disk partition]
path MTU discovery

An algorithm and set of messages used to find the largest packet that can be sent between two endpoints in the network.

pathname

A null-terminated character string starting with an optional slash (/), followed by zero or more directory names separated by slashes, and optionally followed by a filename. If a pathname begins with a slash, it is said to be an absolute pathname, and the path search begins at the root directory. Otherwise, the pathname is said to be a relative pathname, and the path search begins at the current working directory of the process. A slash by itself names the root directory. A null pathname refers to the current working directory.

permanent kernel module

A collection of software that implements a kernel service that must be present at boot time and may not be removed while the system is running. See also kernel loadable module.

persist timer

A timer used by TCP for maintaining output flow on a connection. This timer is started whenever data are ready to be sent, but the send window is too small to bother sending and no data are already outstanding. If no window update is received before the timer expires, a window probe is sent.

physical block

One or more contiguous disk sectors to which the system maps a logical block.

physical mapping (pmap)

The software state, also referred to as the pmap structure, needed to manage the machine-dependent translation and access tables that are used either directly or indirectly by the memory-management hardware. This mapping state includes information about access rights, in addition to address translation.

PID
See [process identifier]
pipe

An interprocess-communication facility that supports the unidirectional flow of data between related processes. Data transfer is stream-oriented, reliable, and flow controlled. A pipe is specified to the shell with the "I" symbol. For example, to connect the standard output of program a to the standard input of program b, the user would type the command "a I b".

pipeline

A collection of processes in which the standard output of one process is connected to the standard input of the next with a pipe.

placement policy

The policy used by the virtual-memory system to place pages in main memory when servicing a page fault. FreeBSD uses page coloring to optimize the placement of pages.

pmap
See [physical mapping]
point-to-point protocol (PPP)

The point-to-point protocol is a line discipline used by networking software to encapsulate and transfer Internet Protocol datagrams over asynchronous serial lines.

See also [line discipline]

polling I/O

The normal mode for a descriptor whereby the system will block if a read request has no data available or a write request has no buffering available. A process can determine whether an I/O operation will block by polling the kernel using the select or poll system call. The select or poll system call can be requested to return immediately with the information or to block until at least one of the requested I/O operations can be completed. non-blocking I/O

See also [signal-driven I/O]

POSIX

The standards group for P1003, the portable operating-system interfaces established by the IEEE. Its first established standard was the kernel interface, 1003.1, which was ratified in 1988. The final POSIX standard was ratified in 1999. Since 1999, the only changes have been to keep the existing POSIX standards current.

PPP
See [point-to-point protocol]
prefetching

The retrieval of data before they are needed. Many machines prefetch machine instructions so that they can overlap the time spent fetching instructions from memory with the time spent decoding instructions.

prepaging

The prefetching of pages of memory. Prepaging is a technique used by virtual-memory systems to reduce the number of page faults.

See also [cluster]

private mapping

When privately mapping a file in virtual memory, changes made to the memory mapping the file are not written back to the mapped file and are not visible to other processes mapping the file.

See also [shared mapping]

probe

The operation of checking to see whether a hardware device is present on a machine. Newer bus designs have a standardized way to identify the devices that are attached to them.

process

In operating systems, a task that contains one or more threads of execution. In UNIX, user processes are created with the fork system call.

process context

The context of a FreeBSD process consists of user-level state, including the contents of its address space and the run-time environment, and kernel-level state, which includes scheduling parameters, resource controls, and identification information. The process context includes everything used by the kernel in providing services for the process.

process group

A collection of processes on a single machine that all have the same process-group identifier. The kernel uses this grouping to arbitrate among multiple jobs contending for the same terminal.

process-group identifier

A positive integer used to identify uniquely each active process group in the system. Process-group identifiers are typically defined to be the PID of the process-group leader. Process-group identifiers are used by command interpreters in implementing job control, when the command interpreter is broadcasting signals with the killpg system call, and when the command interpreter is altering the scheduling priority of all processes in a process group with the setpriority system call.

process-group leader

The process in a process group whose PID is used as the process-group identifier. This process is typically the first process in a pipeline.

process identifier (PID)

A nonnegative integer used to identify uniquely each active process in the system.

process open-file table
See [descriptor table]
processor affinity

In an SMP system, a desire to run a thread on the same processor. For performance reasons, a thread should not be migrated between processors unnecessarily because of the loss of its cached working memory.

processor group

A set of CPU cores on a processor that supports symmetric multithreading or a set of processors in an SMP system that is treated as a unit by the scheduler.

process priority

A parameter used by the kernel to schedule the execution of the threads within a process. The priority for threads running in the timesharing class changes dynamically according to the operation of the thread. In addition, the nice parameter can be set for a process to weight the overall scheduling priority for its threads.

See also [scheduling class]
See also [scheduling priority]

process structure

A data structure maintained by the kernel for each active process in the system. The process structure for a process is always resident in main memory, as opposed to the user structure, which is moved to secondary storage when the process is swapped out.

/proc fllesystem

A filesystem-based interface to active processes that provides process-debugging facilities. Each process is represented by a directory entry in a pseudo-directory named /proc. Applications access the virtual address space of a process by opening the file in /proc that is associated with the process and then using the read and write system calls as though the process were a regular file.

protocol family

A collection of communication protocols, the members of which are related by being part of a single network architecture. For example, the TCP, UDP, IPv4, and ICMPv4 protocols are part of the protocol family for the IPv4 domain.

protocol-switch structure

A data structure that holds all the entry points for a communication protocol supported by the kernel.

pseudo-terminal

A pseudo-terminal provides a software emulation of a hardware terminal. A pseudo-terminal is built from a pair of character devices, a master device and a slave device. The slave device provides a process with an interface identical to that for a hardware terminal. However, instead of having a hardware device driving it, the slave device has another process manipulating it through the master half of the pseudo-terminal. Anything written on the master device is given to the slave device as input, and anything written on the slave device is presented as input on the master device.

PTE
See [page-table entry]
public key encryption

A cryptographic system in which the keys used to encrypt data can be shared publicly, in contrast to systems that require that all keys be kept secret to guarantee the security of the data.

pure demand paging

Demand paging without prepaging.

push migration

When the scheduler actively moves a thread from one CPU to another to balance the computational load in a system.

race condition

A condition in which two or more actions for an operation occur in an undefined order. Trouble arises if there exists a possible order that results in an incorrect outcome.

rapid connection reuse

A new connection that exactly duplicates the addresses and ports of a recently closed stream socket that is in the TIME_WAIT state.

raw-device interface

The character-device interface for block-oriented devices such as disks. This interface provides raw access to the underlying device, arranging for direct I/O between a process and the device.

raw mode
See [canonical mode]
raw socket

A socket that provides direct access to a communication protocol beneath the transport layer. For example, a raw socket in the IPv4 domain gives the user the ability to read and write IP packets directly without using a transport protocol such as UDP or TCP.

read-caching lease

A read-caching lease is issued by an NFS server to a client to allow the client to do data caching but to prohibit the client from modifing the data.

See also [noncaching lease]
See also [write-caching lease]

real GID
See [real group identifier]
real group identifier (real GID)

The GID that is recorded in the accounting record when a process terminates. The real GID for a process is initially set at the time that a user logs into a system and is then inherited by child processes across later fork and exec system calls (irrespective of whether a program is set-group-identifier). saved group identifier

See also [credential]
See also [effective group identifier]
See also [set-group-identifier program]

real UID
See [real user identifier]
real user identifier (real UID)

With respect to a process, the true identity of the user that is running the process. The real UID for a process is initially set at the time a user logs into a system and is then inherited by child processes across later fork and exec system calls (irrespective of whether a program is set-user-identifier). The real UID is recorded in the accounting record when a process terminates. saved user identifier

See also [credential]
See also [effective user identifier]
See also [set-user-identifier program]

receive window

In TCP, the range of sequence numbers that defines the data that the system will accept for a connection. Any data with sequence numbers outside this range that are received are dropped.

See also [sliding-window scheme]

reclaim
See [page reclaim]
reclaim from inactive

A page reclaim from the inactive list. A page can be reclaimed from the inactive list if that page is freed by the page-replacement algorithm, but the page is not reassigned before a process faults on it.

recovery storm

A failure condition that can occur when a server is congested on returning to service after a period of being unavailable. If there is heavy pent-up demand for the server, it may be inundated with requests. If the server simply ignores requests that it cannot handle, the clients will quickly resend them. So the server typically replies "try again later" to the requests that it is not yet ready to service. Clients receiving such a response will wait considerably longer than a typical timeout period before resending their request.

red zone

A read-only region of memory immediately below the last page of the per-thread kernel-mode run-time stack. The red zone is set up by the system so that a fault will occur if a thread overflows the space allocated for its kernel stack.

referenced page

In the virtual-memory system, a page that is read or written.

reference string

A dataset that describes the pages referenced by a process over the time of the process's execution. This description represents the memory-related behavior of the process at discrete times during that process's lifetime.

region

A range of memory that is being treated in the same way. For example, the text of a program is a region that is read-only and is demand paged from the file on disk that contains it.

relative pathname
See [pathname]
reliably-delivered-message socket

A type of socket that guarantees reliable data delivery and preservation of message boundaries and that is not connection based.

relocation

The copying of a program's contents from one place in an address space to another. This copying may be accompanied by modifications to the image of the program so that memory references encoded in the program remain correct after that program is copied. Code that is not bound to a particular starting memory address is said to be relocatable or position independent.

remote procedure call (RPC)

A procedure call made from a client process to invoke a subroutine in a server process. Typically, the client and server processes are running on different machines. A remote procedure call operates much like a local procedure call: the client makes a procedure call, and then waits for the result while the procedure executes.

See also [marshalling]

replacement policy

The policy that a demand-paged virtual-memory-management system uses to select pages for reuse when memory is otherwise unavailable.

resident object

An object that is present in main memory. For example, a page in the virtual address space of a process is resident if its contents are present in main memory.

resident-set size

The number of pages of physical memory held by a process. In a well-tuned system, the resident-set size of a process will be that process's working set. Usually, the precise working set cannot be calculated, so a process will have additional pages beyond that needed for its working set.

retransmit timer

A timer used by TCP to trigger the retransmission of data. This timer is set each time that data are transmitted to a remote host. It is set to a value that is expected to be greater than the time that it will take the receiving host to receive the data and return an acknowledgment.

reverse-mapped page table
See [inverted page table]
root directory

The directory that the kernel uses in resolving absolute pathnames. Each process has a root directory that can be set with the chroot system call, and the system has a unique root directory, the identity of which is set at the time that the system is bootstrapped.

root filesystem

The filesystem containing the root directory that is considered the root of all filesystems on a machine. The identity of a default root filesystem is compiled into a kernel, although the actual root filesystem used by a system may be set to some other filesystem at the time that a system is bootstrapped.

round robin

In queueing, an algorithm in which each requester is serviced for a fixed time in a first-come, first-served order; requests are placed at the end of the queue if they are incomplete after service.

route

In packet-switched-network communication, a route to a destination specifies the host or hosts through which data must be transmitted to reach the destination.

router

A machine, also known as a gateway, that has two or more network interfaces and that forwards packets between the networks to which it is connected. Typically, a router runs a routing process that gathers information on the network topology; it uses that information to devise a set of next-hop routes that it installs in the kernel's routing table. See also ; routing policy.

See also [routing mechanism]

router solicitation

A message sent by a host in an attempt to discover which machine is its router without human intervention. autoconfiguration

See also [neighbor discovery]

routing daemon

A process in FreeBSD that provides a routing-management service for the system. This service uses a protocol that implements a distributed database of routing information that is updated dynamically to reflect changes in topological connectivity.

routing mechanism

The routing facilities included in the kernel that implement externally defined policies. The routing mechanism uses a lookup mechanism that provides a first-hop route (a specific network interface and immediate destination) for each destination.

See also [router]
See also [routing policies]

routing policies

The routing facilities provided in a user-level process that define external policies. Routing policies include all the components that the routing daemon uses in choosing the first-hop routes, such as discovery of the local network topology, implementation of various routing protocols, and configuration information specifying local policies.

See also [router]
See also [routing mechanism]

routing redirect message

A message generated by a router when the latter recognizes that a message that it has received can be delivered via a more direct route.

RPC
See [remote procedure call]
run queue

The queue of those threads that are ready to execute.

See also [idle queue]
See also [sleep queue]

SA
See [security association]
saved GID

A mechanism that records the identity of a setgid program by copying the value of the effective GID at the time that the program is exec'ed. During its execution, the program may temporarily revoke its setgid privilege by setting its effective GID to its real GID. It can later recover its setgid privilege by setting its effective GID back to its saved GID.

See also [credential]
See also [effective group identifier]

saved UID

A mechanism that records the identity of a setuid program by copying the value of the effective UID at the time that the program is exec'ed. During its execution, the program may temporarily revoke its setuid privilege by setting its effective UID to its real UID. It can later recover its setuid privilege by setting its effective UID back to its saved UID.

See also [credential]
See also [effective user identifier]

scatter-gather I/O

Scatter input allows a single read to be placed in several different buffers. Scatter output allows several different buffers to be written in a single atomic write. Scatter-gather I/O uses an iovec structure, an array of buffers and lengths, to identify the buffers to be used for the I/O.

See also [iovec]

scheduling

In operating systems, the planning used to share a resource. For example, process scheduling shares the CPU and main memory.

scheduling class

The FreeBSD kernel has five scheduling classes: kernel interrupts, system calls, real time, time sharing, and idle. Each process is placed into a scheduling class. Within each class, threads of the process are organized by their scheduling priority.

See also [scheduling priority]
See also [process priority]

scheduling priority

A per-process parameter maintained by the kernel that specifies the priority with which the latter will schedule the execution of the threads of the process. When a thread is executing in user mode in the timesharing class, the system periodically calculates the scheduling priority, using the thread priority and the nice parameter.

See also [process priority]
See also [scheduling class]

secondary storage

Storage that holds data that do not fit in main memory. Secondary storage is usually located on rotating magnetic media, such as disk drives.

See also [backing storage]

sector

The smallest contiguous region on a disk that can be accessed with a single I/O operation.

security association (SA)

The basic channel of secure communication in IPSec. Data is secured only in one direction by a security association, which means that two security associations are required to create a fully secure channel between two hosts.

See also [security-parameter index]

security-parameter index (SPI)

A 32-bit piece of data used to identify the end of a security association on a host using IPSec. The security-parameter index is used as a key when working with security associations in a system's security databases.

See also [security association]

segment

A contiguous range of data defined by a base and an extent. In memory management, a segment describes a region of a process's address space. In the TCP protocol, a segment is a range of bytes within a single connection defined by starting and ending sequence numbers.

semaphores

Data structures and a set of functions used for synchronizing access to a shared resource, such as an area of memory. Semaphores implement two functions: a take and a give, such that once one thread has taken a semaphore, all others that follow the first are blocked until the first thread gives the semaphore back.

send window

In TCP, the range of sequence numbers that defines the data that the system can transmit on a connection and be assured that the receiving party has space to hold the data on receipt. Any data with sequence numbers before the start of the send window have already been sent and acknowledged. Any data with sequence numbers after the end of the window will not be sent until the send window changes to include them.

See also [sliding-window scheme]

sense request

A request passed to a communication-protocol module as a result of a process making a stat system call on a socket.

sequenced-packet socket

A type of socket that models sequenced, reliable, unduplicated, connection-based communication that preserves message boundaries.

sequence space

The range of sequence numbers that are assigned to data transmitted over a TCP connection. In TCP, sequence numbers are taken from a 32-bit circular space that starts with an arbitrary value called the initial sequence number.

server process

A process that provides services to client processes via an interprocess-communication facility.

See also [client process]

session

A collection of process groups established for job control purposes. Normally, a session is created for each login shell. All processes started by that login shell are part of its session.

session leader

A process that has created a session. The session leader is the controlling process for the session and is permitted to allocate and assign the controlling terminal for the session. Normally, a session is created for each login shell. All processes started by that login shell are part of its session.

set-group-identifier program

A program that runs with an additional group privilege. Set-group-identifier programs are indicated by a bit in the inode of the file. When a process specifies such a file in an exec system call, the GID of the file is made the effective GID of the process.

set-user-identifier program

A program that runs with an UID different from that of the process that started it running. Set-user-identifier programs are indicated by a bit in the inode of the file. When a process specifies such a file in an exec system call, the UID of the file is made the effective UID of the process.

shadow object

An anonymous object that is interposed between a process and an underlying object to prevent changes made by the process from being reflected back to the underlying object. A shadow object is used when a process makes a private mapping of a file so that changes made by the process are not reflected in the file.

shared mapping

When doing a shared mapping to a file in virtual memory, changes made to the memory mapping the file are written back to the mapped file and are visible to other processes mapping the file. See also private mapping.

shared memory

An area of memory that can be read and written by two different processes. The fastest way to share information between processes on the same system. See also semaphore.

shell

A program that interprets and executes user commands. When a user logs into a UNIX system, a shell process is normally created with its standard input, standard output, and standard error descriptors directed to the terminal or pseudo-terminal on which the user logged in.

short-term-scheduling algorithm

The algorithm used by the system to select the next process to run from among the set of processes that are deemed run-nable. The long-term-scheduling algorithm, on the other hand, can influence the set of runnable processes by swapping processes in and out of main memory (and thus in and out of the set of runnable processes).

signal

In UNIX, a software event. In FreeBSD, this event is modeled after a hardware interrupt.

signal-driven I/O

A mode in which a descriptor can be placed, whereby the system will deliver a SIGIO signal to a process whenever I/O is possible on the descriptor.

See also [nonblocking I/O]
See also [polling I/O]

signal handler

A procedure that is invoked in response to a signal.

signal post

A notification to a process that a signal is pending for that process. Since most of the actions associated with a signal are done by the receiving process, a process that is posting a signal usually does little more than to record the pending signal in the receiving process's process structure and to arrange for the receiving process to be run.

signal-trampoline code

A piece of code that invokes a signal handler. The signal-trampoline code contains instructions that set up parameters for calling a signal handler, do the actual call to the signal handler, and, on return, do a sigreturn system call to reset kernel state and resume execution of the process after the signal is handled.

silly-window syndrome

A condition observed in window-based flow-control schemes in which a receiver sends several small (i.e., silly) window allocations rather than waiting for a reasonable-sized window to become available.

single indirect block
See [indirect block]
slave device

A hardware device that is controlled by a master device. For example, a disk drive is a slave device to a SCSI bus controller. The distinction between master and slave devices is used by the autoconfiguration system. A slave device is assumed to be accessible only if its corresponding master device is present.

sleep queue

The queue of those threads that are blocked awaiting an event. The name is derived from the sleep() routine that places threads on this queue.

See also [idle queue]
See also [run queue]

sliding-window scheme

A flow-control scheme in which the receiver limits the number of data that it is willing to receive. This limit is expressed as a contiguous range of sequence numbers termed the receive window. It is periodically communicated to the sender, who is expected to transmit only those data that are within the window. As data are received and acknowledged, the window slides forward in the sequence space.

See also [receive window]
See also [send window]
See also [sequence space]

small-packet avoidance

In networking, avoiding the transmission of a packet so small that its transmission would be inefficient.

SMP
See [symmetric multiprocessing]
SMT
See [symmetric multithreading]
snapshot

A filesystem snapshot is a frozen image of a filesystem at a given instant in time.

socket

In the FreeBSD interprocess-communication model, an endpoint of communication. Also, the data structure that implements the socket abstraction and the system call that creates a socket.

socket address structure

A generic structure for holding addresses for a socket. Many interprocess communication routines, such as connect() and bind(), need to know the network addresses of the communicating endpoints and require that a socket address structure be passed as a parameter.

soft limit

A limit that may be temporarily exceeded, or exceeded a limited number of times. A soft limit is typically used with a hard limit.

See also [hard limit]

soft link
See [symbolic link]
software interrupt

A software-initiated interrupt. It is requested with an asynchronous system trap.

software-interrupt thread

A thread that is set running in response to a software interrupt. In FreeBSD, input processing for each transport-layer communication protocol is embodied in a software-interrupt thread.

special file
See [device special file]
SPI
See [security-parameter index]
spin mutex

A spin mutex will not relinquish the CPU when it cannot immediately get the requested lock, but it will loop, waiting for the mutex to be released by another CPU.

stack

An area of memory set aside for temporary storage or for procedure and interrupt-service linkages. A stack uses the last-in, first-out (LIFO) concept. On most architectures, the stack grows from high memory addresses to low memory addresses. As items are added to (pushed onto) the stack, the stack pointer decrements; as items are retrieved from (popped off) the stack, the stack pointer increments.

stack segment

A segment that holds a stack.

See also [bss segment]
See also [data segment]
See also [text segment]

stale translation

A translation or mapping that was previously true, but is no longer valid. For example, on machines that have a translation lookaside buffer, if a page-table entry in memory is changed to alter the mapping, any address translation for that page that is present in the translation lookaside buffer must be flushed to avoid a stale translation.

standalone device driver

A device driver that is used in a standalone program. A standalone device driver usually differs from a device driver used in an operating system in that it does not have interrupt services, memory management, or full support for virtual-memory mapping. In the FreeBSD standalone I/O library, for example, a standalone device driver polls a device to decide when an operation has completed and is responsible for setting up its own memory mapping when doing transfers between the device and main memory.

standalone I/O library

A library of software that is used in writing standalone programs. This library includes standalone device drivers that are used to do I/O.

standalone program

A program that can run without the support of an operating system.

standard error

The I/O stream on which error messages are conventionally placed. This stream is usually associated with descriptor 2 in a process.

standard input

The I/O stream on which input is conventionally received. This stream is usually associated with descriptor 0 in a process.

standard output

The I/O stream to which output is conventionally directed. This stream is usually associated with descriptor 1 in a process. start routine A device-driver routine that is responsible for starting a device operation after the system has acquired all the resources that are required for the operation.

stateless server

A server that does not need to maintain any information about which clients it is serving or which data have been passed to them. Every request that is received by such a server must be completely self-contained, providing all information needed to fulfill it.

sticky bit

The bit in an inode representing a directory that shows that an unprivileged user may not delete or rename files of other users in that directory. The sticky bit may be set by any user on a directory that the user owns or for which she has appropriate permissions. Historically, the bit in an inode that indicated that the text segment of the program was to be shared and kept memory or swap-space resident because of expected future use. That bit is no longer needed for this use because the virtual-memory system tracks recently used executables.

stream socket

A type of socket that models a reliable, connection-based byte stream that can support out-of-band data transmission.

superblock

A data structure in the on-disk filesystem that specifies the basic parameters of the filesystem.

superuser

The user whose UID is 0. Processes owned by the superuser are granted special privileges by UNIX. The superuser's login name is usually root.

swap area

A region on secondary storage that is used for swapping and paging.

swap device

A device on which a swap area resides.

swapper

In FreeBSD, the name of the kernel process that implements the swapping portion of the memory-management facilities. Historically, the swapper is process 0.

See also [pagedaemon]

swapping

A memory-management algorithm in which entire processes are moved to and from secondary storage when main memory is in short supply.

swap space
See [swap area]
symbolic link

A file whose contents are interpreted as a pathname when it is supplied as a component of a pathname. Also called a soft link.

See also [hard link]

symmetric cryptography

A cryptographic system that uses the same key to encrypt data as it does to decrypt data, sometimes referred to as secret key cryptography.

See also [asymmetric cryptography]

symmetric multiprocessing (SMP)

A multiprocessor consists of two or more CPUs connected to a common main memory. Symmetric multiprocessing describes a kernel that can run simultaneously on all the CPUs at the same time.

See also [symmetric multithreading]

symmetric multithreading (SMT)

A processor that has multiple CPU cores on which threads can be run is said to support symmetric multithreading. Each core acts as a processor itself, but all cores on the CPU share the same cache memory.

synchronous

Synchronized with the currently running process. For example, in UNIX, all I/O operations appear to be synchronous: The read and write system calls do not return until the operation has been completed. (For a write, however, the data may not really be written to their final destination until some time later—for example, in writing to a disk file.)

system activity

An entry into the kernel. System activities can be categorized according to the event or action that initiates them: system calls, hardware interrupts, hardware traps, and software-initiated traps or interrupts.

system call

In operating systems, a request to the system for service; also called a system service request.

system clock

The device that maintains the system's notion of time of day. On most systems, this device is an interval timer that periodically interrupts the CPU. The system uses these interrupts to maintain the current time of day and to do periodic functions such as thread scheduling.

system mode
See [kernel mode]
tags

An extensible system for adding arbitrary data to an mbuf or mbuf cluster for communicating information between different modules in the network stack without having to modify the packet data.

TCB
See [thread control block]
TCP
See [transmission control protocol]
terminal

In computer systems, a device used to enter and receive data interactively from a computer. Most terminals include a CRT, which displays data that are received from a computer. In the Electrical Industry Association (EIA) standard RS-232-C for connecting computers and data-terminal equipment (DTE), a terminal is a device that is placed at the other end of a wire that is connected to data-communications equipment (DCE). In this standard, a terminal might be any type of device, rather than only a device on which people type.

termios structure

The structure used to describe terminal state. Terminal state includes special characters, such as the erase, kill, and word-erase characters; modes of operation, such as canonical or noncanonical; and hardware serial-line parameters, such as parity and baud rate.

text segment

The segment of a program that holds machine instructions. The system usually makes a program's text segment read-only and shareable by multiple processes when the program image is loaded into memory.

See also [bss segment]
See also [data segment]
See also [stack segment]

thrashing

A condition where requested memory utilization far exceeds the memory availability. When a machine is thrashing, it usually spends more time doing system-related tasks than executing application code in user mode.

thread

The unit of execution of a process. A thread requires an address space and other resources, but it can share many of those resources with other threads. Threads sharing an address space and other resources are scheduled independently and can all do system calls simultaneously.

thread control block (TCB)

A data structure used to hold thread context. The hardware-defined TCB contains the hardware portion of this context. The software TCB contains the software portion and is located in memory immediately after the hardware TCB.

tick

An interrupt by the system clock.

time quantum

In a timesharing environment, the period of time that the process scheduler gives a process to run before it preempts that process so that another process can execute. Also called a time slice.

timer backoff

The rate at which a timer value is increased. For example, in TCP, the value of the retransmit timer is determined by a table of multipliers that provide a near-exponential increase in timeout values.

time slice
See [time quantum]
time-stable identifier

An identifier that refers uniquely to some entity both while it exists and for a long time after it is deleted. A time-stable identifier allows a system to remember an identity across transient failures and to detect and report errors for attempts to access deleted entities.

TLB
See [translation lookaside buffer]
top half

With regard to system operation, the routines in the kernel that are invoked synchronously as a result of a system call or trap. These routines depend on per-process state and can block by calling sleep().

See also [bottom half]

trace trap

A trap used by the system to implement single-stepping in program debuggers. On architectures that provide trace-bit support, the kernel sets the hardware-defined trace bit in the context of the thread being debugged and places the thread on the run queue. When the thread next runs, the trace bit causes a trap to be generated after the thread executes one instruction. This trap is fielded by the kernel, which stops the thread and returns control to the debugging process.

track

In computer systems, the sectors of a disk that are accessible by one head at one of its seek positions.

track cache

When the kernel is reading from a disk, memory associated with the disk that holds data that are passing under the disk heads regardless of whether they have been requested explicitly. When the kernel is writing to a disk, memory associated with the disk in which data are stored until the disk heads reach the correct position for writing them.

translation lookaside buffer (TLB)

A processor cache containing translations for recently used virtual addresses.

transmission control protocol (TCP)

A connection-oriented transport protocol used in the Internet. TCP provides for the reliable transfer of data, as well as for the out-of-band indication of urgent data.

transport layer

The layer of software in the network subsystem that is responsible for moving data between two sockets. Depending on the type of transport protocol used, the data may be delivered as a sequenced, in-order stream or as an unordered set of individual messages. See also , unreliable datagram protocol.

See also [transmission control protocol]

transport mode

One of two modes used for secure communications in IPSec. In transport mode only the payload of a packet is protected, whereas the network layer protocol header is left exposed.

See also [tunnel mode]

triple indirect block
See [indirect block]
tty driver

The software module that implements the semantics associated with a terminal or pseudo-terminal device.

See also [line discipline]

tunnel mode

One of two modes used for secure communication in IPSec. In tunnel mode, the packet to be secured is completely contained within another packet that carries the inner packet between two endpoints. The endpoints are the boundaries of the tunnel.

See also [transport mode]

2MSL timer

A timer used by the TCP protocol during connection shutdown. The name refers to the fact that the timer is set for twice the maximum time that a segment may exist in the network. This value is chosen to ensure that future connections will not mistakenly accept late messages from an older connection.

See also [maximum segment lifetime]

type-ahead

Transmission of data to a system, usually by a user typing at a keyboard, before the data are requested by a process.

UDP
See [user datagram protocol]
UID
See [user identifier]
uio

A data structure used by the system to describe an I/O operation. This structure contains an array of iovec structures; the file offset at which the operation should start; the sum of the lengths of the I/O vectors; a flag showing whether the operation is a read or a write; and a flag showing whether the source and destination are both in the kernel's address space or whether the source and destination are split between user and kernel address spaces.

See also [iovec]

update dependency

The required ordering of related updates to separate metadata structures to ensure recoverability in the presence of unpredictable failures. An update that must be done later has an update dependency on an earlier update because it cannot be done until the earlier update is committed to stable storage.

See also [metadata]

urgent data

In TCP, data that are marked for urgent delivery.

user datagram protocol (UDP)

A simple, unreliable datagram protocol used in the Internet protocols. UDP provide peer-to-peer, multicast and broadcast addressing and optional data checksums A single version of UDP works the same way on top of both IPv4 and IPv6.

user identifier (UID)

A nonnegative integer that identifies a user uniquely. UIDs are used in the access-control facilities provided by the filesystem. saved user identifier

See also [credential]
See also [effective user identifier]
See also [real user identifier]
See also [set-user-identifier program]

user mode

The least-privileged processor-access mode. User processes run in user mode.

See also [kernel mode]

user-request routine

A set of routines provided by each communication protocol that directly supports a socket (a protocol that indirectly supports a socket is layered underneath a protocol that directly supports a socket). These routines serve as the main interface between the layer of software that implements sockets and the communication protocol. The interprocess-communication facilities make calls to the user-request routines for most socket-related system calls.

See also [connect request]
See also [control request]
See also [listen request]
See also [sense request]

user structure

A data structure maintained by the kernel for each active process in the system. The user structure contains the process statistics and signal actions. For compatibility with older debuggers, much of the process state is copied to the user structure before generating a core dump. Unlike the process structure, the user structure for a process is moved to secondary storage if the process is swapped out. Also referred to as the u-dot area and user area.

virtual address

An address that references a location in a virtual address space.

virtual-address aliasing

Two or more processes mapping the same physical page at different virtual addresses. When using an inverted page table, there can only be one virtual address mapping any given physical page at any one time. Here, the kernel must invalidate the page-table entry for the aliased page whenever it switches between the processes with the conflicting virtual addresses for that page.

See also [inverted page table]

virtual address space

A contiguous range of virtual-memory locations.

virtual machine

A machine whose architecture is emulated in software.

virtual memory

A facility whereby the effective range of addressable memory locations provided to a process is independent of the size of main memory; that is, the virtual address space of a process is independent of the physical address space of the CPU.

virtual-memory object

A kernel data structure that represents a repository of data—for example, a file. An object contains a pager to get and put the data from and to secondary storage, and a list of physical pages that cache pieces of the repository in memory.

vnode

An extensible object-oriented interface containing generic information about a file. Each active file in the system is represented by a vnode, plus filesystem-specific information associated with the vnode by the filesystem containing the file. The kernel maintains a single systemwide table of vnodes that is always resident in main memory. Inactive entries in the table are reused on a least-recently used basis.

wait

The system call that waits for the termination of a descendant process.

wait channel

A value used to identify an event for which a thread is waiting. In most situations, a wait channel is defined as the address of a data structure related to the event for which a thread is waiting. For example, if a thread is waiting for the completion of a disk read, the wait channel is specified as the address of the buffer data structure supplied to the disk I/O system.

wildcard route

A route that is used if there is no explicit route to a destination.

window probe

In TCP, a message that is transmitted when data are queued for transmission, the send window is too small for TCP to bother sending data, and no message containing an update for the send window has been received in a long time. A window-probe message contains a single octet of data.

wired page

Memory that is not subject to replacement by the pageout daemon. A nonpageable range of virtual addresses has physical memory assigned when the addresses are allocated. Wired pages must never cause a page fault that might result in a blocking operation. Wired pages are typically used in the kernel's address space.

word-erase character

The character that is recognized by the terminal handler in canonical mode to mean "delete the most recently typed word on this terminal." By default, preceding whitespace and then a maximal sequence of nonwhitespace characters are erased. Alternatively, an alternate erase algorithm tuned to deleting pathname components may be specified. Each terminal session can have a different word-erase character, and the user can change that character at any time with an tcsetattr system call. The terminal handler does not recognize the word-erase character on terminals that are in non-canonical mode.

See also [erase character]
See also [kill character]

working directory
See [current working directory]
working set

The set of pages in a process's virtual address space to which memory references have been made over the most recent few seconds. Most processes exhibit some locality of reference, and the size of their working set is typically less than one-half of their total virtual-memory size.

write-caching lease

A write-caching lease is issued by an NFS server to a client to allow the client to cache reads and writes for the period of the lease.

See also [noncaching lease]
See also [read-caching lease]

zombie process

A process that has terminated but whose exit status has not yet been received by its parent process (or by init).


FreeBSD Kernel Internals on Video

The course is based on this book and provides a firm background of the FreeBSD kernel. It covers all the topics in this book. In addition, it covers other related topics including performance measurement, system tuning, and security issues. The class consists of twelve 160-minute (2-hour, 40-minute) lectures on the FreeBSD kernel. There are assigned readings to be completed before viewing each lecture. The first ten lectures have a set of exercises to be done after each video is viewed. Exercise answer sheets are provided along with follow-up comments on the exercises at the beginning of the lecture following the one in which they are assigned.

The syllabus for the the course is as follows:

  1. Introduction: kernel terminology and basic kernel services

  2. Processes: process structure and process management

  3. Virtual memory: virtual-memory management, paging, and swapping

  4. Kernel-resource management: scheduling and signals

  5. Introduction to I/O: special files and pseudo-terminal handling

  6. Kernel I/O structure: multiplexing I/O, autoconfiguration strategy, and structure of a disk device driver

  7. Filesystem overview: filesystem services and the block I/O system (buffer cache)

  8. Filesystem implementation: local filesystem implementation, support for multiple filesystems, and the network filesystem (NFS)

  9. Interprocess communication: concepts and terminology, basic IPC services, and code review of a simple application that demonstrates use of the IPC and network facilities

  10. Networking implementation: system layers and interfaces, Internet protocols (TCP/IP), and routing issues

  11. System tuning: performance measurement and system tuning

  12. Other topics: crash dump analysis and security issues

In addition to the twelve lecture videos, you also receive a copy of the course notes, which consist of copies of all overhead slides used in the course, a set of weekly readings from this textbook, exercises (along with answers) for each lecture, and a set of papers that provide supplemental reading to the text.

The cost of the course is $1,300 for individuals, $2,300 for companies who wish to buy it for employee use. Discounts are available for full-time students and people who make a plausible case of financial hardship.

Courses shipped before December 2005 are based on the previous edition of this book, The Design and Implementation of the 4.4BSD Operating System. Courses shipped after December 2005 are based on this edition. For up-to-date information on course status or to place an order, see the Web page at

http://www.mckusick.com/courses/


Advanced FreeBSD Course on Video

This course provides an in-depth study of the source code of the FreeBSD kernel. It is aimed at users who already have a good understanding of the algorithms used in the FreeBSD kernel and want to learn the details of the algorithm's implementation. Students are expected to have either taken a FreeBSD Kernel Internals class (such as the one described on the previous page) or to have throughly read and understood this book. They are also expected to have a complete background in reading and programming in the C programming language. Students will not need to prove relationship with a source license holder, as the course is based on the non-proprietary kernel sources released by the FreeBSD project.

The class consists of fifteen 160-minute (2-hour, 40-minute) lectures on the FreeBSD kernel source code. The lecture topics are:

  1. Organization, overview of source layout

  2. Kernel header files

  3. System calls and file open

  4. Pathname translation and file creation

  5. Vnode interface mechanics, write to a local file

  6. Opening, using, and closing locally connected sockets

  7. User datagram protocol and routing

  8. TCP Algorithms

  9. Fork, exit, and exec

  10. Signal generation and delivery, scheduling

  11. Virtual memory header files, file mapping

  12. Page fault service, pageout processing

  13. NFS client and server operation

  14. Multiplexing with select, system startup

  15. Special topics: filesystem layering and soft updates

In addition to the fifteen lecture videos, you also receive a CD-ROM with a copy of the FreeBSD kernel source covered in the lectures plus a printed copy of the lecture notes and selected FreeBSD kernel source files highlighted in the lectures.

The cost of the course is $1,500 for individuals, $2,500 for companies who wish to buy it for employee use. Discounts are available for full-time students and people who make a plausible case of financial hardship.

Courses shipped before June 2006 are based on FreeBSD 4.0. Courses shipped after June 2006 are based on the stable release of FreeBSD 5.X as of January 2006. For up-to-date information on course status or to place an order, see the Web page at:

http://www.mckusick.com/courses/


CSRG Archive CD-ROMs

Thanks to the efforts of the volunteers of the "UNIX Heritage Society" (see http://www.tuhs.org) and the willingness of Caldera to release 32/V under an open source license (see http://www.mckusick.com/csrg/calder-lic.pdf), it is now possible to make the full source archives of the University of California at Berkeley's Computer Systems Research Group (CSRG) available.

The archive contains four CD-ROMs with the following content:

CD-ROM #l-Berkeley Systems 1978-1986

Ibsd

2.9pucc

4.1. snap

4.2buglist

2.10

2bsd

4.1a

4.3

2.79

3bsd

4.1c.l

VM.snapshot.l

2.8

4.0

4.1c.2

pascal.2.0

2.9

4.1

4.2

pascal.2.10

CD-ROM #2-Berkeley Systems 1987-1993

4.3reno

4.4BSD-Litel

net.l

4.3tahoe

VM.snapshot.2

net.2

CD-ROM #3-Final Berkeley Releases

4.4

4.4BSD-Lite2

CD-ROM #4-Final /usr/src including SCCS files

Contrib

admin

games

local

sys

Makefile

bin

include

old

usr.bin

README

contrib

lib

sbin

usr.sbin

SCCS

etc

libexec

share

The University of California at Berkeley wants you to know that these CD-ROMs contain software developed by the University of California at Berkeley and its many contributors.

The CD-ROMs are produced using standard pressing technology, not with write-once CD-R technology. Thus, they are expected to have a 100-year lifetime rather than the 10-20 years expected of CD-R disks. The CDs are sold only in complete sets; they are not available individually. The price for the 4-CD set is $99. The archive can be ordered from

http://www.mckusick.com/csrg/

The compilation of this archive is copyright © 1998 by Marshall Kirk McKusick. You may freely redistribute it to anyone else. However, I appreciate you buying your own copy to help cover the costs that I incurred in producing the archive.


History of UNIX at Berkeley

The history of UNIX development at Berkeley has been recounted in detail by Marshall Kirk McKusick in a two-and-a-half-hour video in which he brings the history of UNIX to life, complete with anecdotes and interesting personal stories.

Marshall Kirk McKusick's work with BSD development spans some 25 years. His first paper was on the implementation of Berkeley Pascal in 1979. It was followed by his pioneering work in the 1980s on the BSD Fast File System, the BSD virtual memory system, and the final release from the UC Berkeley Computer Systems Research Group, 4.4 BSD Litc2. Since 1993, he has been working on FreeBSD, adding soft updates, snapshots, and the second-generation Fast Filesystem to the system. A key figure in BSD development, his experiences chronicle not only the innovative technical achievements but also the interesting personalities and philosophical debates in UNIX over the past 25 years. His talk covers many of these areas as he describes his years at the Computer Systems Research Group at Berkeley.

The price for the video is $49. The video can be ordered from

http://www.mckusick.com/history/


BSD Daemon Shirts

The John Lasseter daemon, first commissioned for 4.2BSD, is now available printed on T-shirts and embroidered on Polo shirts. Available are both the original standing daemon and the 4.4BSD daemon scampering after the glowing orb with his trident. The T-shirt displays the text "May the Source Be With You" under the picture. The polo shirts have just the picture embroidered using over 18,000 stitches. All shirts are 100% pre-shrunk cotton. Shirts are available in a range of colors and sizes; check the Web site for current availability.

The cost for T-shirts is $15 each; the cost for polo shirts is $35 each. A history of the BSD daemon and ordering information is available at

http://www.mckusick.com/beastie/


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

absolute pathname 2nd 3rd 4th
accept system call 2nd 3rd 4th 5th
definition
access control list 2nd 3rd 4th
access control, filesystem
access rights 2nd
receiving
access system call
access vnode operator
accounting, process resource
accton
ACL [See access control list]
active page list
address family 2nd 3rd
address resolution protocol 2nd
purpose of
address space [See virtual address space]
address structure
Internet
local domain
socket 2nd
address translation 2nd
adjtime system call
advanced programmable interrupt controller 2nd
advisory locking 2nd 3rd
advlock vnode operator
AF_INET 2nd
AF_INET6
AH [See authentication header]
aio_read system call 2nd
aio_return system call
aio_suspend system call
aio_waitcomplete system call
aio_write system call, 2nd
algorithm
elevator sorting
for disksort()
for physical I/O
allocation
descriptor
FFS file block 2nd
inode
kernel memory
process identifier
allocdirect structure 2nd
allocindir structure
ancillary data 2nd 3rd 4th
Andrew Filesystem
anonymous object 2nd 3rd
AOUT executable format
API [See application programming interface]
append-only file
Apple OS/X operating system 2nd 3rd
application programming interface
application, client-server
arguments, marshalling of 2nd
ARP [See address resolution protocol]
ARPANET
arpintr()
arpresolve()
assembly language in the kernel 2nd 3rd 4th
AST [See asynchronous system trap]
asymmetric cryptography 2nd
asynchronous system trap 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th
AT&T 2nd
ATA [See ATA disk]
ATA disk
ATTA CHED flag
definition
attribute manipulation, filesystem
attribute update, filestore
authentication data
authentication header 2nd
autoconfiguration 2nd 3rd 4th 5th
contribution of
device driver support for


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

B programming language
b_to_q() 2nd
Babaoglu, Özalp
background process 2nd 3rd 4th
backing storage 2nd
basic input-output system
bawrite()
bcopy()
BCPL
bdwrite()
benefit of global vnode table
bind system call
definition
biod
biodone()
BIOS [See basic input-output system]
bitmap dependencies, soft updates
blkatoff vnode operator
block clustering 2nd
block size 2nd 3rd
bmsafemap structure 2nd 3rd
Bolt Beranek and Newman
boot 2nd 3rd
flags
boot_time
bootstrapping 2nd 3rd 4th
setting time when, [See also boot]
bottom half of 2nd
device driver
terminal driver
bqrelse()
bread() 2nd
break character
breakpoint fault 2nd
brelse()
bremfree()
broadcast message 2nd
address
bss segment 2nd
buf structure,
bufdaemon 2nd
buffer cache 2nd
4.4BSD
consistency
effectiveness
interface
memory allocation
buffer list
CLEAN 2nd
DIRTY
EMPTY
LOCKED 2nd
buffer update
buffer wait 2nd
buffering
policy, protocol
bus_driver_added()
bus_probe_nomatch()
bus_read_ivar()
bus_write_ivar()
bwrite() 2nd
bzero()


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

C library
system calls in the
C programming language 2nd
C-block
C-list 2nd 3rd
cache
buffer 2nd
page list
vnode
callback 2nd 3rd 4th 5th
callout queue 2nd 3rd
CAM [See common access method]
canonical mode 2nd 3rd
canrecurse flag
castor oil
catq()
caught signal 2nd 3rd
CD-ROM 2nd
CD9660 filesystem
cdevsw structure 2nd
character device
driver
ioctl
operations
character device interface 2nd 3rd
character processing
chdir system call 2nd 3rd
checksum 2nd 3rd
chflags system call 2nd
child process 2nd 3rd 4th
chkdq()
chmod system call 2nd
Chorus operating system
chown system call 2nd
chroot system call 2nd 3rd
classless inter-domain routing 2nd 3rd
CLEAN buffer list 2nd
client process 2nd
client server
application
model
programming
clock
alternate
interrupt rate
real-time
clock_skew 2nd
cloning route 2nd 3rd
close system call 2nd 3rd 4th 5th 6th 7th 8th 9th
close vnode operator
closedir()
clustering, block 2nd
clustering, page 2nd 3rd 4th 5th
cold start 2nd
common access method 2nd
communication domain 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th
data structures
COMPLETE flag 2nd 3rd 4th 5th
definition
Computer Systems Research Group 2nd
config 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th
configuration
file 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th
congestion window 2nd 3rd
connect request 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th
connect system call 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th
definition
connection
queueing, socket 2nd
shutdown, TCP
contents update, filestore
context switching 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th
involuntary 2nd
thread state
continuation style 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th
control request 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th
control-output routine, protocol
controlling process 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th
controlling terminal 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th 30th 31st 32nd 33rd 34th 35th 36th 37th 38th 39th 40th 41st
revocation of
cooked mode
copy object, 4.4BSD
copy-on-write 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th
core file 2nd 3rd
coredump()
cpu_exit()
cpu_switch()
crash dump 2nd 3rd 4th 5th
crash, system 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th
create vnode operator 2nd
creation and deletion, filestore
credential 2nd 3rd 4th 5th 6th 7th
cron
crypto_done()
crypto_freesession()
crypto_invoke()
crypto_proc()
crypto_register()
cryptographic descriptor
cryptography
asymmetric 2nd
session
symmetric 2nd
csh shell
CTSS operating system
current working directory 2nd 3rd 4th
cursig() 2nd
cv_broadcast()
cv_signal()
cv_timedwait()
cv_timedwait_sig()
cv_wait()
cv_wait_sig()
cv_waitq_remove()


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

daemon
process 2nd
routing
DARPA [See Defense Advanced Research Projects Agency]
data segment 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th 30th
expansion
data structures
communication domain
data-communications equipment 2nd
data-terminal equipment
data-terminal ready
datagram socket 2nd
DCE [See data-communications equipment]
dead filesystem
deadlock
avoidance during fork system call
avoidance when locking resources
detection
memory
debugging
gdb 2nd
information in exec header
process
decapsulation 2nd 3rd
decision, local-remote
default pager
Defense Advanced Research Projects Agency 2nd 3rd 4th
steering committee
definition
ATTACHED flag
COMPLETE flag
DEPCOMPLETE flag
defrtrlist_update()
delayed write 2nd 3rd
demand paging [See paging]
denial of service attack
DEPCOMPLETE flag 2nd 3rd 4th 5th
definition
descriptor 2nd
allocation
table 2nd 3rd
design
4.2BSD IPC
FreeBSD IPC 2nd 3rd
network
devd
device
flags 2nd
interrupt
interrupt handler
number 2nd
pager
probing
special file 2nd
swap 2nd
device driver 2nd 3rd
bottom half of
code for select system call
interrupt handling
maximum transfer size
sections of a
support for autoconfiguration
support for select system call
top half of
devinfo
df
diradd structure 2nd
DIRCHG flag
direct memory access 2nd
direct route
directory 2nd 3rd
entry 2nd 3rd
table 2nd 3rd
dirrem structure 2nd
DIRTY buffer list
disk
device interface
device operations
label
partition 2nd 3rd 4th 5th
disksort()
algorithm for
distributed filesystem
distributed program
DMA [See direct memory access]
DNS [See domain name system]
doadump()
domain [See communication domain]
domain name system
double indirect block 2nd 3rd
dtom()
dump 2nd 3rd
live
dumpsys()
dup system call 2nd 3rd 4th
implementation of
dup2 system call 2nd 3rd


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

EAGAIN system error 2nd 3rd 4th 5th
effective GID [See effective group identifier]
effective group identifier 2nd
effective UID [See effective user identifier]
effective user identifier 2nd
Eighth Edition UNIX
EINTR system error 2nd 3rd
elevator sorting algorithm 2nd
ELF executable format
Elz, Robert 2nd 3rd
EMPTY buffer list
encapsulating security protocol, IPSec
encapsulating-security payload 2nd 3rd
encapsulation 2nd 3rd
environment, location of process
epoch
erase character 2nd
errno 2nd 3rd 4th 5th
ESP [See encapsulating-security payload]
ether_demux()
Ethernet 2nd 3rd
event handler
eviction notice 2nd
exec header
exec system call 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th
operation of
exit system call 2nd 3rd 4th 5th
operation of
status 2nd 3rd
exit()
exported filesystem services
extattrctl system call
extension header 2nd 3rd
external data representation


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

fast filesystem
cluster map
cylinder group
ffs_balloc()
ffs_read()
ffs_write()
file block allocation 2nd
file block extension
fragment-descriptor table 2nd
free-space reserve 2nd 3rd 4th
local allocation routines
fault rate 2nd
fchdir system call 2nd
fchflags system call 2nd
fchmod system call 2nd
fchown system call 2nd
fcntl system call 2nd 3rd
fdesc filesystem
Federal Information Processing Standard
fetch policy 2nd
FFS [See fast filesystem]
fhopen system call
fifo 2nd
file 2nd 3rd
access validation
append-only
control, filesystem
deactivation
executable
handle, NFS 2nd
hole in 2nd
immutable
interpretation, filesystem
management, filesystem
offset 2nd 3rd
permission bits
size distribution
file block
allocation, FFS 2nd
locality of reference
file entry
flag 2nd
handling during fork system call
implementation of
object oriented 2nd
operations
file locking
NFS
file structure 2nd 3rd
file-table flag
filename
negative caching of
whiteout
filestore
attribute update
contents update
creation and deletion
size update
filesystem 2nd
/proc 2nd 3rd
3BSD 2nd
4.2BSD
4.3BSD
4.4BSD 2nd
access control
attribute manipulation
CD
distributed
fdesc
file control
file interpretation
file management
initialization
ISO
kernfs
name creation
name deletion
name translation 2nd 3rd
NT
operator, valloc
operator, vfree
operator, vget
portal 2nd
procfs
quotas
support for multiple
umapfs 2nd
find
First Edition UNIX
first-level bootstrap
flock system call
foreground process 2nd 3rd
fork system call 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th
deadlock avoidance during
file entry handling during
forward 2nd
forward-mapped page table 2nd
forwarding mechanism
free page list
free() 2nd 3rd
freeblks structure 2nd 3rd
FreeBSD
as a real-time system 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th
goals
IPC design 2nd 3rd
kernel, division of software in
portability of
freefile structure
fsck 2nd 3rd 4th 5th 6th
dependencies, soft updates
fstat system call 2nd
fsync system call 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th
fsync vnode operator
ftp
ftruncate system call
futimes system call


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

g_down
g_up
gateway 2nd
gdb 2nd
generation number 2nd
GENIE operating system
GEOM [See geometry layer]
geometry layer 2nd 3rd
getattr vnode operator
getc()
getdirentries system call
getfsstat system call
gethostbyname()
gethostbyname2 library call definition
gethostbyname2()
getlogin system call
getnewbuf()
getnewvnode() 2nd
getpeername system call 2nd
getrusage system call
getsockname system call 2nd
getsockopt system call 2nd 3rd 4th
global page-replacement algorithm 2nd
global vnode table, benefit of
goals, FreeBSD
Greenwich time [See Universal Coordinated Time]
group identifier 2nd 3rd 4th 5th 6th 7th 8th
use in file-access validation
gsignal()


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

handle_written_inodeblock()
hard limit 2nd
hard link
hardclock() 2nd
hardware_cache_fetch
Harris, Guy
header prediction, TCP 2nd
heap 2nd
high watermark on
socket 2nd 3rd
terminal
history of
job control
process management
home directory 2nd 3rd
hop limit 2nd
hop-by-hop option
host identifier
host name
host unreachable
message 2nd


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

I/O
nonblocking 2nd 3rd 4th 5th 6th 7th 8th
redirection 2nd
scatter/gather 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th 30th
signal driven 2nd 3rd
I/O buffer
I/O stream, 33
ICMP [See Internet control message protocol]
icmp_error()
icmp_input()
idempotent 2nd
idle
loop 2nd
process 2nd
queue
swap time
IEEE. and [See Institute of Electrical]
if_data structure
if_done()
if_output() 2nd 3rd
if_start()
ifaddr structure 2nd 3rd
ifconfig
ifnet structure 2nd
ignored signal
imgact
immutable file
implementation of
dup system call
file entry
ioctl system call
pipe
inactive page list 2nd
inactive vnode operator 2nd
inactive, reclaim from
inbound 2nd
indirect route
init 2nd 3rd 4th 5th 6th 7th
initial sequence number 2nd
initialization
filesystem
virtual memory, [See also bootstrapping]
initiate_write_inodeblock()
inode 2nd 3rd
allocation
contents
number 2nd 3rd 4th 5th 6th 7th 8th
inodedep structure
insecure mode
Institute of Electrical and Electronic Engineers 2nd
integrity-check value 2nd
interactive program 2nd
Interdata 8/32
interface
buffer cache
character device 2nd 3rd
disk device
line switch 2nd
International Organization for Standardization
filesystem
model
protocol suite 2nd
Internet addresses
structure
Internet control message protocol 2nd 3rd 4th 5th 6th 7th 8th
interaction with routing
Internet domain 2nd
Internet Engineering Task Force 2nd
Internet key exchange 2nd
Internet protocol 2nd 3rd 4th 5th 6th 7th 8th 9th
packet forwarding
pseudo-header
Internet service providers
interpreter 2nd
interprocess communication 2nd 3rd 4th 5th 6th 7th 8th
design, 4.2BSD
design, FreeBSD 2nd 3rd
message queue 2nd
reliable delivery
shared memory
interprocessor interrupt
interrupt
device
interrupt handling
device driver
interruptible sleep() 2nd
interval time
inverted page table
involuntary context switching 2nd
ioctl system call 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd
implementation of
ioctl vnode operator
ioctl, character device
iovec structure 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th 30th
ip_forward()
ip_input()
ip_output() 2nd
ip6_output()
IPC_NOWAIT
IPI [See interprocessor interrupt]
IPSec 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th 30th 31st 32nd 33rd 34th
authentication header
encapsulating security protocol
ipsec_common_input()
ipsec4_process_packet()
IPv6 2nd 3rd 4th 5th
ISA bus
ISO [See International Organization for Standardization]
issignal(), operation of
ITS operating system


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

jailkill system call
job 2nd
job control 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th
history of
signals in FreeBSD
terminal driver support for 2nd 3rd
use of process group 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th
Joy, William


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

KAME 2nd 3rd
keepalive packet
keepalive timer
Kerberos authentication
KERNCONF
kernel 2nd
assembly language in the 2nd 3rd 4th
loadable modules 2nd
loading of
memory allocation
mode 2nd 3rd
partitioning, reason for
preemption
process 2nd
return from
security level
state 2nd
kernfs filesystem
key 2nd
kill character 2nd
kill system call
killpg system call 2nd
kmem_alloc()
kmem_alloc_pageable()
kmem_alloc_wait()
kmem_free()
kmem_free_wakeup()
kmem_malloc()
kqueue
kse_create system call
ktrace system call


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

LAN [See local-area network]
lbolt 2nd
lchmod system call
lchown system call
lease
NFS
noncaching
obtaining an, NFS
read-caching 2nd
write-caching
least recently used
library, shared
lightweight process 2nd
limits in system 2nd
line discipline 2nd 3rd
PPP
line mode 2nd
line switch interface 2nd
linesw structure 2nd
link layer 2nd
link system call 2nd
link vnode operator
Linux operating system 2nd 3rd 4th 5th 6th 7th
LISP programming language
listen request 2nd
listen system call 2nd 3rd
definition
Lite, 4.4BSD 2nd
live dump
llinfo_nd6
ln_hold
load average
local domain 2nd 3rd
local page-replacement algorithm 2nd
local-area network 2nd 3rd
local-remote decision
locality of reference 2nd
lock vnode operator
LOCKED buffer list 2nd
locking
advisory 2nd 3rd
mandatory 2nd
resources on a shared-memory multiprocessor
resources, deadlock avoidance when
socket data buffer
lockinit()
logical block 2nd
login 2nd
login name
login shell
long-term scheduling algorithm
lookup vnode operator 2nd
lost+found
low watermark on
socket
terminal 2nd
LRU [See least recently used]
lseek system call 2nd 3rd
lstat system call
lutimes system call


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

m_adj()
m_copy()
m_copydata() 2nd
m_copym()
M_EXT
m_free()
m_freem()
m_get()
m_gethdr()
m_hdr structure
M_PKTHDR
M_PREPEND()
m_pullup() 2nd
Mach operating system 2nd 3rd 4th 5th 6th 7th 8th 9th
Macklem, Rick
magic number 2nd
main memory 2nd
major device number 2nd
make
malloc() 2nd 3rd 4th 5th 6th 7th
management information base
mandatory access control
mandatory locking 2nd
MAP_SHARED
mapping
structure 2nd
marshalling of arguments 2nd
Massachusetts Institute of Technology 2nd
maximum transmission unit 2nd 3rd 4th
maximum_lease_term 2nd
maxusers
mb_alloc()
mbuf 2nd
design, 4.4BSD
memory allocation
buffer cache
kernel
memory deadlock
memory management
hardware, VAX
portability of
system 2nd
memory overlay
memory-management unit 2nd 3rd 4th
merged from current
message queue 2nd
System V
metadata 2nd 3rd 4th 5th 6th 7th 8th
metrics, route 2nd
MFC [See merged from current]
mi_startup()
MIB [See management information base]
MINIX operating system
minor device number 2nd
mkdir structure
mkdir system call 2nd 3rd 4th
mkdir vnode operator
MKDIR_BODY flag
MKDIR_PARENT flag
mkfifo system call
mknod system call
mknod vnode operator
mlock system call 2nd
definition of
mmap system call 2nd 3rd 4th 5th 6th 7th 8th 9th
definition of
mmap vnode operator
MMU [See memory-management unit]
modem control
ignored
mount 2nd
mount options
mount system call 2nd 3rd 4th 5th 6th
mountd
mprotect system call 2nd 3rd
definition of
msgsnd system call
msq_mtx
msync system call 2nd
definition of
mtod()
MTU [See maximum transmission unit]
mtx_destroy()
mtx_lock()
mtx_trylock()
mtx_unlock()
multicast
Multics operating system 2nd
multilevel feedback queue 2nd
multiprocessor
locking resources on a shared-memory
virtual memory for a shared-memory
munlock system call
definition of
munmap system call 2nd 3rd 4th 5th
definition of
mutex, network thread
mutex, spin


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

Nagle, John
name
creation, filesystem
deletion, filesystem
login
translation, filesystem 2nd 3rd
named object
National Bureau of Standards
nd_input()
nd6_na_input()
nd6_timer()
negative caching of filename
neighbor discovery 2nd 3rd
Net1 release
Net2 release
NetBSD 2nd 3rd
network
additions in FreeBSD
architecture 2nd
byte order
design
layer 2nd
mask
queue limiting
thread mutex
time protocol
timer 2nd 3rd
Network Disk Filesystem
Network Filesystem 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th 30th 31st 32nd 33rd 34th 35th 36th 37th 38th
asynchronous writing
delayed writing
file handle 2nd
file locking
hard mount
interruptible mount
lease
lease, obtaining an
overview
recovery storm 2nd
soft mount
network interface
layer 2nd
newblk structure
newbus 2nd
newfs 2nd 3rd
next header
nextc()
NFS [See Network Filesystem]
nfsd 2nd
nice 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th 30th 31st
Ninth Edition UNIX
nonblocking I/O 2nd 3rd 4th 5th 6th 7th 8th
noncaching lease
nonlocal goto
Not-Quite Network Filesystem 2nd 3rd 4th
Novell 2nd
NQNFS [See Not-Quite Network Filesystem]
NT filesystem
null modem connection


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

O_ASYNC flag
object
oriented file entry 2nd
shadow 2nd 3rd
virtual memory
octet
Olson, Arthur
open system call 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th
open vnode operator
open-file entry
OpenBSD 2nd 3rd 4th 5th 6th
opendir()
optimal replacement policy 2nd
orphaned process group 2nd
OS/X operating system, Apple, 3 2nd
OSF/1
OSI [See International Organization for Standardization]
out-of-band data 2nd
receipt of
transmission of
overlay 2nd


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

packet
filter 2nd 3rd
forwarding, IP
page cache 2nd
page clustering 2nd 3rd 4th 5th
page fault 2nd 3rd 4th
page lists
active
cache
free
inactive 2nd
wired
page replacement
4.4BSD 2nd
in the VMS operating system
page table 2nd
forward-mapped 2nd
pages 2nd 3rd
page, wired 2nd 3rd 4th 5th
page-table entry 2nd 3rd 4th
pagedaemon 2nd 3rd 4th 5th 6th 7th 8th 9th 10th
pagedep structure
pagein()
pageout daemon [See pagedaemon]
pageout in progress
pager
device
swap
vnode
pagezero 2nd
paging 2nd 3rd 4th 5th 6th
systems, characteristics of
panic 2nd [See also system crash]
parent directory 2nd
parent process 2nd 3rd 4th
path
path MTU discovery 2nd 3rd
pathname 2nd 3rd
paths through network node
PDP- 2nd 3rd 4th 5th 6th 7th
peripheral component interconnect
permanent kernel module 2nd
persist timer
personal computer 2nd 3rd 4th
stack growth on
PF_KEY
address extension
association extension
base header
PF_KEY_V2
pfctlinput()
pgo_alloc()
pgo_dealloc()
pgo_getpage()
pgo_haspage() 2nd
pgo_init()
pgo_putpages()
physical block 2nd
physical I/O
algorithm for
physical mapping
physio() 2nd
PID [See process identifier]
ping 2nd 3rd
pipe 2nd 3rd 4th 5th
implementation of
system call 2nd 3rd 4th
pipeline 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th
placement policy 2nd
Plan 2nd
pmap
functions
initialization
module 2nd
structure 2nd
pmap_change_wiring() 2nd
pmap_clear_modify() 2nd
pmap_clear_ptes()
pmap_clear_reference() 2nd
pmap_copy_page() 2nd
pmap_enter()
pmap_is_modified() 2nd
pmap_page_protect()
pmap_pinit() 2nd
pmap_protect()
pmap_qenter() 2nd
pmap_qremove() 2nd
pmap_release() 2nd
pmap_remove()
pmap_remove_all()
pmap_ts_referenced()
pmap_zero_page() 2nd
point-to-point protocol
poll interface, System V
poll system call 2nd 3rd
poll vnode operator
pollfd structure
polling I/O 2nd
portability of
FreeBSD
memory management
Seventh Edition UNIX
portable operating system interface 2nd 3rd 4th 5th 6th
portal filesystem 2nd
postsig()
PPP [See point-to-point protocol]
ppp
pr_ctlinput() 2nd 3rd 4th
pr_ctloutput() 2nd 3rd
pr_drain() 2nd
pr_fasttimo() 2nd
pr_input()
pr_output() 2nd
pr_pfil() 2nd
pr_slowtimo() 2nd
pr_usrreqs() 2nd
preemption
kernel
thread 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th
prefetching
prefix option
prepaging 2nd
prison structure
private mapping 2nd 3rd
probe
process 2nd 3rd
context 2nd
debugging
flags
kernel 2nd
lightweight 2nd
open-file table 2nd
profiling 2nd
resource accounting
scheduling 2nd
state, change of 2nd
structure 2nd 3rd
termination
virtual address space
virtual time
process group 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th
association with, socket 2nd
hierarchy
identifier 2nd 3rd 4th
job-control use of 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th
leader
orphaned 2nd
terminal 2nd 3rd
process identifier 2nd 3rd 4th 5th
allocation
process management
history of
process priority 2nd 3rd
calculation of
processor affinity 2nd
processor group 2nd
procfs filesystem
profclock() 2nd
profil system call
profiling
process 2nd
timer 2nd
program relocation 2nd
programming language
B
BCPL
C 2nd
LISP
protocol 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th
buffering policy
control-output routine
switch
switch structure 2nd 3rd
protocol family 2nd 3rd
protosw structure
pru_abort()
pru_accept()
pru_connect2()
pru_control()
pru_disconnect()
pru_fasttimo()
pru_listen()
pru_peeraddr()
pru_rcvd()
pru_rcvoob()
pru_send() 2nd
pru_sense()
pru_shutdown()
pru_slowtimo()
pru_sockaddr()
ps
ps_strings structure
pseudo-header, IP
pseudo-terminal 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th 30th 31st 32nd 33rd 34th 35th 36th 37th
psignal()
ptrace system call
public key encryption
pure demand-paging 2nd
push migration
putc()
pv_entry structure 2nd


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

q_to_b()
queue limiting, network
quota.group 2nd
quota.user 2nd
quotacheck 2nd
quotactl system call
quotas
contribution of
format of record 2nd
limits 2nd


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

race condition 2nd 3rd 4th 5th
radix search trie
range lock, System
rapid connection reuse 2nd
raw device
interface 2nd
raw mode 2nd
raw socket 2nd 3rd 4th
output processing
read system call 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th
read-caching lease 2nd
READ_
readdir vnode operator
readdir()
readlink vnode operator
readv system call 2nd
real GID [See real group identifier]
real group identifier 2nd
real UID [See real user identifier]
real user identifier 2nd
real-time clock
real-time system, FreeBSD as a 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th
real-time timer 2nd
reboot system call 2nd
receive window 2nd
reclaim from inactive
reclaim vnode operator 2nd
recovery storm, NFS 2nd
recvfrom system call 2nd
recvit()
recvmsg system call 2nd 3rd
data structures for
red zone 2nd 3rd
reference string 2nd
region 2nd
relative pathname 2nd 3rd 4th
reliably-delivered-message socket 2nd
Remote Filesystem
remove vnode operator
rename system call 2nd 3rd
addition of
rename vnode operator
replacement policy 2nd
request for comments 2nd 3rd 4th
resetpriority() 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th
resident-set size 2nd
resource
accounting, process
limit
utilization
retransmit timer
return from kernel
reverse-mapped page table 2nd 3rd
revocation of controlling terminal
revoke system call 2nd 3rd 4th 5th
rewinddir()
RFC [See request for comments]
rfork system call
rip_input()
Ritchie, Dennis
rlimit structure
rm
rmdir system call 2nd 3rd 4th
rmdir vnode operator
root directory
root filesystem 2nd 3rd
root user 2nd
round robin 2nd
round-trip time 2nd
RPC timeout
roundrobin() 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th
route metrics 2nd
route, cloning 2nd 3rd
routed
router 2nd 3rd
advertisement
entry
solicitation 2nd
routing
daemon 2nd 3rd
information protocol
interaction with ICMP
mechanism
policy 2nd
redirect 2nd
socket
rpc.lockd 2nd 3rd
rpc.statd 2nd
RS-232 serial line 2nd 3rd
rtalloc()
rtentry
structure
rtfree
rtprio system call
rtredirect() 2nd
RTT [See round-trip time]
run queue 2nd 3rd
runq_add()
runq_choose()
operation of
runq_remove()


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

SA [See tunnel mode;security association]
Samba
Santa Cruz Operation
savecore
saved GID 2nd
saved UID 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th
sblock()
sbrk system call 2nd 3rd 4th
sbunlock()
SC22 WG15
scatter/gather I/O 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th 30th
sched_balance()
sched_lock
schedcpu() 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th
schednetisr()
scheduler()
scheduling 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th
class 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th
long-term algorithm
parameters
priority 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th
process 2nd
short-term algorithm
thread
secondary storage 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th
secure mode
security
association
association, transport mode
association, tunnel mode
level, kernel
parameter index 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th 30th 31st
seekdir()
segment 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th
bss 2nd
data 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th 30th
stack 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th
text 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th
select system call 2nd 3rd 4th
device driver code for
device driver support for
motivation for, –
seltrue()
selwakeup()
semaphores 2nd 3rd
System
semctl system call
semget system call 2nd
semop system call 2nd
send system call 2nd 3rd
send window
sendfile system call
sendit()
sendmsg system call 2nd 3rd
data structures for
sendsig()
sendto system call 2nd 3rd
sense request 2nd
sequence space
sequenced packet socket 2nd
server message block
server process 2nd
session 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th
leader 2nd
set-group-identifier program 2nd
set-user-identifier program 2nd
setattr vnode operator
seteuid system call
setlogin system call
setpriority system call
setrunnable() 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th 30th
setsid system call
setsockopt system call 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th
settimeofday system call
Seventh Edition UNIX
portability of
sh shell 2nd 3rd
shadow object 2nd 3rd
shared library
shared mapping 2nd
shared memory 2nd 3rd
System V 2nd 3rd
shared text segment
shell
csh
login
sh 2nd 3rd
shmat system call
shmdt system call 2nd
shmem system call 2nd
shmget system call
short-term scheduling algorithm 2nd
shutdown system call 2nd 3rd 4th
SI_ORDER_FIRST 2nd
SI_ORDER_SECOND 2nd
sigaction system call 2nd 3rd
SIGALRM
sigaltstack system call
SIGCHLD 2nd 3rd
SIGCONT 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th
SIGHUP 2nd 3rd
SIGINT
SIGIO 2nd 3rd 4th
SIGKILL 2nd 3rd
signal 2nd 3rd
checking for a pending
comparison with other systems
driven I/O 2nd 3rd
handler 2nd 3rd 4th
masking
priority
restrictions on posting
stack 2nd
trampoline code 2nd
sigpause system call
sigprocmask system call 2nd
SIGPROF 2nd
sigreturn system call 2nd 3rd
SIGSTOP 2nd 3rd
sigsuspend system call
sigtramp()
SIGTSTP 2nd
SIGTTIN
SIGTTOU 2nd
SIGURG
SIGVTALRM
SIGWINCH
silly-window syndrome 2nd
single indirect block 2nd 3rd
size update, filestore
sleep queue 2nd
sleep() 2nd 3rd 4th 5th 6th 7th 8th 9th
interruptible 2nd
operation of
use of sleep()
sliding-window scheme
small computer system interface
bus
small-packet avoidance 2nd
SMP [See symmetric multiprocessing]
snapshot
SO_LINGER
socantrcvmore()
SOCK_STREAM
sockaddr_dl
sockaddr_in
sockaddr_in6
socket 2nd 3rd 4th 5th 6th 7th 8th 9th
address structure 2nd
connection queueing 2nd
data buffer locking
data buffering 2nd 3rd
error handling
options
process group association with 2nd
state transitions during rendezvous
state transitions during shutdown
states 2nd
types 2nd
socket system call 2nd 3rd 4th 5th 6th 7th
definition
socketpair system call 2nd
soconnect()
soft limit 2nd
soft link 2nd 3rd [See also symbolic link]
soft updates
bitmap dependencies
fsck dependencies
softclock()
softdep_disk_io_initiation()
softdep_disk_write_complete()
softdep_update_inodeblock()
software interrupt 2nd
sohasoutofband()
soisconnected()
soisdisconnected()
solisten()
sonewconn()
soreceive() 2nd 3rd
sorwakeup()
sosend() 2nd 3rd 4th
source-quench processing, TCP
Spec
special file 2nd 3rd
special-device
spin mutex
ssh 2nd 3rd
stack
growth on PC
segment 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th
zero filling of user
stale data
stale translation
standalone
device driver 2nd
I/O library 2nd
program
standard error 2nd
standard input 2nd
standard output 2nd
start routine 2nd
stat structure 2nd 3rd
stat system call 2nd 3rd 4th 5th 6th
statclock()
stateless protocol 2nd
statfs system call
statistics collection
sticky bit 2nd
stop character
strategy vnode operator
strategy()
stream I/O system
stream socket 2nd
STREAMS 2nd
su
Sun Microsystems 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th
superblock 2nd
superuser 2nd 3rd
supplementary group array
swap
area 2nd
device 2nd
out
out, 4.4BSD
pager
pager, 4.4BSD
partitions
space 2nd 3rd
swapin()
swapoff
swapper 2nd 3rd 4th 5th
swapping 2nd 3rd
in FreeBSD, reasons for
swi_sched() 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th
swp_pager_async_iodone()
symlink system call
symlink vnode operator
symmetric cryptography 2nd
symmetric multiprocessing 2nd 3rd 4th 5th 6th 7th
symmetric multithreading 2nd
sync system call 2nd
syncer 2nd 3rd 4th
syscall()
sysctl system call 2nd 3rd
syslogd
system activity 2nd
system call 2nd
handling 2nd
system calls
accept 2nd 3rd 4th 5th
access
adjtime
aio_read 2nd
aio_return
aio_suspend
aio_waitcomplete
aio_write 2nd
bind
chdir 2nd 3rd
chflags 2nd
chmod 2nd
chown 2nd
chroot 2nd 3rd
close 2nd 3rd 4th 5th 6th 7th 8th 9th
connect 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th
dup 2nd 3rd 4th
dup2 2nd 3rd
exec 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th
exit 2nd 3rd 4th 5th
extattrctl
fchdir 2nd
fchflags 2nd
fchmod 2nd
fchown 2nd
fcntl 2nd 3rd
fhopen
flock
fork 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th
fstat 2nd
fsync 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th
ftruncate
futimes
getdirentries
getfsstat
getlogin
getpeername 2nd
getrusage
getsockname 2nd
getsockopt 2nd 3rd 4th
ioctl 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd
jailkill
kill
killpg 2nd
kse_create
ktrace
lchmod
lchown
link 2nd
listen 2nd 3rd
lseek 2nd 3rd
lstat
lutimes
mkdir 2nd 3rd 4th
mkfifo
mknod
mlock 2nd 3rd
mmap 2nd 3rd 4th 5th 6th 7th 8th 9th 10th
mount 2nd 3rd 4th 5th 6th
mprotect 2nd
msgsnd
msync 2nd
munlock
munmap 2nd 3rd 4th 5th
open 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th
pipe 2nd 3rd 4th
poll 2nd 3rd
profil
ptrace
quotactl
re voke 2nd 3rd 4th 5th
read 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th
readv 2nd
reboot 2nd
recvfrom 2nd
recvmsg 2nd 3rd
rename 2nd 3rd
rfork
rmdir 2nd 3rd 4th
rtprio
sbrk 2nd 3rd 4th
select 2nd 3rd 4th
semctl
semget 2nd
semop 2nd
send 2nd 3rd
sendfile
sendmsg 2nd 3rd
sendto 2nd 3rd
seteuid
setlogin
setpriority
setsid
setsockopt 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th
settimeofday
shmat
shmdt 2nd
shmem 2nd
shmget
shutdown 2nd 3rd 4th
sigaction 2nd 3rd
sigaltstack
sigpause
sigprocmask 2nd
sigreturn 2nd 3rd
sigsuspend
socket 2nd 3rd 4th 5th 6th 7th
socketpair 2nd
stat 2nd 3rd 4th 5th
statfs
symlink
sync 2nd
sysctl 2nd 3rd
tcsetattr 2nd 3rd
truncate 2nd 3rd
undelete 2nd
unlink 2nd 3rd 4th
unmount 2nd
utimes 2nd
vfork 2nd 3rd 4th 5th
wait 2nd 3rd 4th 5th 6th 7th 8th
wait4 2nd
write 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th
writev 2nd
system crash 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th
system error, EAGAIN 2nd 3rd 4th 5th
system error, EINTR 2nd 3rd
system performance 2nd 3rd 4th 5th 6th 7th 8th
system startup
initial state
System V 2nd 3rd
message queue
poll interface
range lock
semaphores
shared memory 2nd 3rd
terminal driver


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

#! 2nd
.. 2nd 3rd 4th 5th 6th
/
/dev 2nd 3rd
/dev/console
/dev/fd
/dev/kmem
/dev/mem 2nd 3rd
/dev/null
/etc
/etc/defaults/rc.conf
/etc/exports
/etc/gettytab
/etc/group
/etc/master.passwd
/etc/rc 2nd
/etc/rc.conf
/etc/ttys
/kern
/proc filesystem 2nd 3rd
/sys/kern/sched_4bsd.c
/sys/kern/sched_ule.c
/tmp
/usr/sbin/config
/var/quotas 2nd
/var/run/lock
3BSD
filesystem 2nd
4.0BSD
4.1BSD 2nd
4.2BSD 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th
filesystem
IPC design
virtual-memory interface
4.3BSD 2nd 3rd 4th 5th
filesystem
Tahoe release
4.4BSD 2nd 3rd 4th
buffer cache
copy object
filesystem 2nd
Lite 2nd
mbuf design
page replacement 2nd
supported architectures
swap out
swap pager
virtual memory


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

table, forward-mapped page 2nd
tags 2nd
tasklist 2nd
tcp_close()
tcp_ctloutput()
tcp_delack()
tcp_input() 2nd 3rd
tcp_output()
operation of
tcp_usr_send()
tcp_usr_shutdown()
tcsetattr system call 2nd 3rd
tcsetpgrp()
TDF_NEEDRESCHED 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th
telldir()
telnet 2nd
TENEX operating system
Tenth Edition UNIX
terminal 2nd 3rd
multiplexer
output
terminal driver
bottom half of
data queues 2nd
hardware state
modem transitions
modes 2nd
open()
software state
special characters 2nd
System V
top half of
user interface
window size 2nd
terminal process group 2nd 3rd
termios structure 2nd
text segment 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th [See also shared text segment]
Thompson, Ken 2nd
thrashing 2nd
thread 2nd 3rd
control block 2nd 3rd
mutex, network
preemption 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th
priority
priority, calculation of
priority, while sleeping
queues
scheduling
state
state, change of
thread_exit()
tick 2nd
time
interval
of day
of-day register
process virtual
quantum 2nd
representation
slice 2nd 3rd
stable identifier 2nd
time zone handling
timer
2MSL
network 2nd 3rd
profiling 2nd
real-time 2nd
resolution of
virtual-time 2nd
watchdog
top half of 2nd
device driver
terminal driver
TOPS-20 operating system
trace trap 2nd
traced process 2nd
track cache
translation lookaside buffer
transmission control protocol 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th
connection shutdown
data buffering
delayed acknowledgments in 2nd
handling of urgent data
header prediction 2nd
implementation, use of 4BSD
receive window
retransmission handling
source-quench processing
window updates
transport layer 2nd
transport mode 2nd
security association
trap handling 2nd
trap type code
trap()
triple indirect block 2nd 3rd
truncate system call 2nd 3rd
addition of
truncate vnode operator
ttread()
ttselect()
ttstart() 2nd
ttwakeup()
ttwrite()
2nd [See terminal driver]
ttyclose()
ttylclose()
ttymodem()
ttyoutput()
ttypoll()
Tunis operating system 2nd
tunnel mode 2nd
security association
tunneling
type-ahead 2nd
types of


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

UFS 2nd 3rd
ufs_bmap() 2nd
UFS1 2nd
UFS2 2nd
uio structure 2nd 3rd
uiomove() 2nd 3rd
uma_zalloc()
uma_zcreate()
uma_zfree()
uma_zone_set_max()
umapfs filesystem 2nd
undelete system call 2nd
Universal Coordinated Time 2nd
universal serial bus
University of California at Berkeley
UNIX
Programmer's Manual
System
System Laboratory
System V, Release 2nd
United Filesystem
unlink system call 2nd 3rd 4th
unmount
unmount system call 2nd
unputc()
update dependency 2nd
update vnode operator 2nd
updatepri()
ureadc()
urgent data 2nd
TCP handling of
transmission, styles of
urgent delivery
USENET 2nd
user datagram protocol 2nd 3rd 4th 5th 6th 7th
user identifier 2nd 3rd 4th 5th 6th 7th
use in file-access validation
user mode 2nd 3rd
user request routines
user structure 2nd 3rd
user-request routine
UTC.


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

V Kernel operating system
valloc filesystem operator
VAX
memory management hardware
vfork system call 2nd 3rd 4th 5th
implementation issues
operation of,
vfree filesystem operator
vfs.usermount
vget filesystem operator
vgone()
virtual address space 2nd
process
virtual memory 2nd
4.4BSD
advantages of
cache coherency
change protection
change size
for a shared-memory multiprocessor
initialization
interface, 4.2BSD
object
usage calculation of
virtual private network
virtual-address aliasing 2nd 3rd
virtual-time timer 2nd
vm_daemon() 2nd
vm_fault() 2nd 3rd
vm_forkproc()
vm_map structure 2nd
vm_map_entry structure 2nd 3rd 4th 5th 6th 7th 8th 9th 10th
vm_object structure 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th
vm_page structure 2nd 3rd 4th 5th 6th 7th
vm_page_alloc()
vm_page_io_finish()
vm_page_startup()
vm_pageout()
vm_pageout_scan()
VMS operating system
page replacement in the
vmspace structure 2nd
vmspace_exec()
vmspace_fork()
vmspace_free()
vnlru 2nd
vnode 2nd 3rd 4th 5th
cache
vnode operator
access
advlock
blkatoff
close
create 2nd
fsync
getattr
inactive 2nd
ioctl
link
lock
lookup 2nd
mkdir
mknod
mmap
open
poll
readdir
readlink
reclaim 2nd
remove
rename
rmdir
setattr
strategy
symlink
truncate
update 2nd
write
vnode pager
vnode_pager_setsize()
vop_access_args structure


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

wait channel 2nd 3rd
wait system call 2nd 3rd 4th 5th 6th 7th 8th 9th
wait4 system call 2nd
operation of
wakeup() 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th 30th
operation of
wakeup_one()
watchdog timer
whiteout, filename
wildcard route 2nd
window probe
window size 2nd
window system 2nd [See also X Window System]
wired page 2nd 3rd 4th 5th
definition of
list
word-erase character 2nd
working set 2nd
worklist structure 2nd
workstation
write system call 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th
write vnode operator
write-caching lease
write_slack 2nd
writev system call 2nd


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

X Window System 2nd
X.25
X/OPEN 2nd
XDR [See external data representation]
XINU operating system
xterm 2nd


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

zalloc()
zero filling of user stack
zfree()
zombie process 2nd 3rd
zone, red 2nd 3rd